System and method for monitoring a grid of hosting resources in order to facilitate management of the hosting resources

ABSTRACT

Some embodiments provide a method. The method interfaces with several nodes of a hosting system. Each node includes resources for hosting several virtual machines. The method collects statistics for facilitating deployment of the virtual machines across the several resource nodes from the resource nodes. The method determines a particular node from the several nodes with sufficient available resources to host a configuration for a virtual machine based on the collected statistics. The method deploys the configuration for the virtual machine to the particular node for hosting on the particular node.

CLAIM OF BENEFIT TO PRIOR APPLICATION

This patent application claims the benefit of the U.S. ProvisionalPatent Application 61/099,254, entitled “System and Method for AutomatedAllocation and Configuration of Hosted Services”, filed Sep. 23, 2008,the U.S. Provisional Patent Application 61/140,838, entitled “System andMethod for Automated Allocation and Configuration of Hosted Services”,filed Dec. 24, 2008, the U.S. Provisional Patent Application 61/140,835,entitled “Method and Apparatus for Adapting and Hosting a Configurationof a Computer System”, filed Dec. 24, 2008, the U.S. Provisional PatentApplication 61/145,962, entitled “System and Method for AutomatedAllocation and Configuration of Hosted Services”, filed Jan. 20, 2009,the U.S. Provisional Patent Application 61/145,965, entitled “Method andApparatus for Adapting and Hosting a Configuration of a ComputerSystem”, filed Jan. 20, 2009, the U.S. Provisional Patent Application61/159,437, entitled “System and Method for Automated Allocation andConfiguration of Hosted Services”, filed Mar. 11, 2009, the U.S.Provisional Patent Application 61/159,438, entitled “Method andApparatus for Adapting and Hosting a Configuration of a ComputerSystem”, filed Mar. 11, 2009, and the U.S. Provisional PatentApplication 61/165,900, entitled “System and Method for AutomatedAllocation and Configuration of Hosted Services”, filed Apr. 1, 2009.All of the above enumerated Provisional Patent Applications areincorporated herein by reference.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application is related to the following applications: U.S. patentapplication Ser. No. 12/421,597, entitled “System and Method forAutomated Allocation of Hosting Resources Controlled by DifferentHypervisors”, filed Apr. 9, 2009; U.S. patent application Ser. No.12/421,599, entitled “System and Method for Automated Criteria BasedDeployment of Virtual Machines Across a Grid of Hosting Resources”,filed Apr. 9, 2009; U.S. patent application Ser. No. 12/421,602,entitled “System and Method for Automated Configuration of HostingResources”, filed Apr. 9, 2009; U.S. patent application Ser. No.12/421,604, entitled “Automated System and Method to Customize andInstall Virtual Machine Configurations for Hosting in a HostingEnvironment”, filed Apr. 9, 2009; U.S. patent application Ser. No.12/421,605, entitled “Automated System and Method to Provision andAllocate Hosting Resources”, filed Apr. 9, 2009; and U.S. patentapplication Ser. No. 12/421,608, entitled “Automated System and Methodfor Managing Hosting Resources”, filed Apr. 9, 2009.

FIELD OF THE INVENTION

The present invention relates to virtualized hosting services.

BACKGROUND OF THE INVENTION

Hosting services provide a means whereby multiple users can implementcustom server configurations without the overhead costs associated withpurchasing, upgrading, and maintaining equipment needed to implement theconfiguration. Instead, a hosting service provider maintains andprovisions a grid of hardware nodes that are shared amongst the multipleusers. More specifically, resources of a single node can be partitionedand each of these partitions can be allocated to host a virtual serverconfiguration of a different user.

Each hardware node in the grid includes one or more processing units(e.g., a CPU, multiple CPUs, CPUs with multiple processing cores, ASICs,graphics processing units, etc.), memory, block devices (e.g., diskstorage devices), networking capabilities, and other such computingresources that often exceed the computing resources required by anysingle user's server configuration. By partitioning the resources of ahardware node amongst multiple server configurations, a hosting serviceprovider minimizes the amount of hardware needed to host all such serverconfigurations, while maximizing the usage of the available hardware.

Virtualization provides the means for partitioning the hardwareresources amongst the multiple server configurations. Virtualizationcreates the façade that each server configuration is individually hostedon dedicated equipment with a particular set of resources. Two or moreserver configurations are provided non-conflicting sets of resources ofthe same hardware node such that a guaranteed amount of processingresources is available to each such configuration. In other words, asingle physical resource is partitioned to operate as multiple logicalresources.

As shown in FIG. 1, virtualization allows a single computing device 110the ability to function as two or more different computing devices witheach device having distinct sets of hardware resources and softwareresources. For instance, configuration 120 may be allocated 40% of thememory and 80% of the processor cycles of the device 110 andconfiguration 130 may be allocated the remaining 60% of the memory and20% of the processor cycles of the device 110. Additionally, theconfiguration 120 may operate using a first operating system with afirst set of configuration parameters and the configuration 130 mayoperate using a second operating system with a second set ofconfiguration parameters.

An added benefit of virtualization is that a failure in oneconfiguration does not disrupt the operation of the otherconfigurations, even though all such configurations operate overphysical resources of a single device. With reference to FIG. 1, shouldthe configuration 120 crash due to an improper configuration of theoperating system, the configuration 130 will continue operatingunhindered as the resources used by each configuration 120 orconfiguration 130 operate independent of one another.

At the core of each virtualization solution is the hypervisor. Thehypervisor, also referred to as the virtual machine monitor, manages alogical partitioning of a physical set of hardware resources of aphysical device or node between different virtualized guests. Eachvirtualized guest implements one or more virtual machines over a logicalpartition. The hypervisor partitions underlying hardware resources suchthat each virtual machine is provided what logically appears as adistinct and unshared set of hardware resources. However, the hypervisormaps the virtual machine hardware calls to a corresponding subset ofphysical hardware resources that are actually shared by all virtualmachines operating on a particular hardware node. The hypervisor is thusresponsible for mapping the hardware resources of a node to a set ofvirtual resources. The set of virtual resources can then be distributedindependently to one or more operating systems or applications thattogether form the one or more virtual machines. In this manner, eachvirtual machine effectively is provided its own resources (e.g., aprocessor, memory, disk storage, networking, etc.), and the operatingsystem of each virtual machine operates with little to no change overthe provided set of resources.

Different vendors implement hypervisors differently (e.g., Xen®,Parallels®, VMware®, Kernel Virtual Machine® (KVM), etc.). Two prominenthypervisor types are defined as “type 1” hypervisors and “type 2”hypervisors. FIG. 2 illustrates a type 1 hypervisor 210 of a prior artvirtualization implementation. In this figure, the type 1 hypervisor210, is located at a layer below domain 0 (Dom0) 225, where Dom0 225 isthe first domain or virtual machine started by the hypervisor on boot.Other virtual machines that execute one or more different guestoperating systems and one or more applications in conjunction with thetype 1 hypervisor 210 are enumerated as Dom1 through DomN 230.

As shown, the type 1 hypervisor 210 is located between the various Domsand the hardware resources 220. The type 1 hypervisor 210 controlsaccess to the hardware resources 220. Accordingly, there is only asingle layer of abstraction between the type 1 hypervisor 210 and thehardware resources 220.

FIG. 3 illustrates a type 2 hypervisor 310 with a host operating system(OS) as Dom0 320. The host OS at Dom0 320 operates at a layer betweenthe type 2 hypervisor 310 and the hardware resources 330. In thisconfiguration, Dom0 320 is provided special access to the hardwareresources 220 while Doms 1-n are unprivileged domains that by defaultare not provided direct access to the hardware. In this figure, thereare two layers of abstraction between the type 2 hypervisor 310 and thehardware resources 330 since the host OS 320 directly controls thehardware resources 330 of the electronic device. Therefore, the type 2hypervisor 310 utilizes a combination of host OS calls and directhardware calls to facilitate the logical partitioning of the hardwareresources 330 of the electronic device in order to allow for thesimultaneous and non-conflicting operation of multiple virtual machinesrepresented as Doms 1-n 340.

In some instances, a type 1 hypervisor more efficiently performs a setof operations issued by a virtual machine over the hardware resources ofa node than a type 2 hypervisor. In other instances, a type 2 hypervisormore efficiently performs the set of operations issued by the virtualmachine over the hardware resources of a node than a type 1 hypervisor.Each virtualization vendor may implement type 1, type 2, or otherhypervisor types differently from other vendors. Each particular vendormay utilize a distinct application programming interface (API) with adistinct set of commands in order to provide virtualization. Therefore,even hypervisors of the same type that are implemented differently bydifferent vendors may have different performance parameters.

Even though current virtualization solutions (i.e., hypervisors) allowhosting service providers the ability to better utilize their hardwareresources on individual nodes, these solutions nevertheless lack thefunctionality needed to create a diverse multi-node hosting service gridoperable with multiple different hypervisors. Each hypervisor of theprior art virtualization solutions operates independent of otherhypervisors and other nodes. The calls and interfaces of the varioushypervisors are not cross-compatible. Therefore, a hosting service isunable to provide a unified grid of nodes that leverages the flexibilityand efficiencies of each of the various different hypervisorimplementations.

FIG. 4 illustrates proprietary hypervisor managers 410 and 420implemented by some prior art virtualization solutions to automate theinteraction with one or more hypervisors 430 and 440 across the nodegrid 450. As noted above, these proprietary hypervisor managers 410 and420 only interface with a single hypervisor from a single virtualizationsoftware provider. Therefore, each proprietary manager 410 and 420 isadministered separately from the others. Such separate administrationpartitions the node grid 450 rather than unifies the grid. Additionally,such separate administration does not provide a unified view of theresources of the grid. Instead, each proprietary manager provides adisjointed view that makes it difficult to monitor the grid resourceusage and availability of resources over the entire grid of nodes.

A further shortcoming of current virtualization solutions is that thehypervisor and other virtualization components do not provide customsoftware configurations to automatedly tailor a virtual serverconfiguration per user specifications. In fact, the goal of mosthypervisors is to permit the independent operation of the virtualmachine such that the virtual machine is wholly unaware of thevirtualization occurring underneath. Current hypervisors avoid accessingand modifying the file system of the virtual machine. Accordingly,current hypervisors circumvent the ability to provide automated hosting,requiring system administrators to manually establish and deploy virtualmachine configurations.

To attempt to alleviate this issue, some hosting service providersemploy a grid management system to retrieve and copy static softwareimages from an image store to the resources allocated for the virtualmachine configuration. However, this manner of configuration restrictsthe ability of hosting service providers to customize theconfigurations. Specifically, a static set of configurations areavailable and a “one-size-fits-all” approach is taken to satisfy therequirements of multiple different users. Therefore, after each install,a system operator manually accesses each virtual machine in order toadjust the configurations to create the modified or customconfiguration. Such manual operation increases the time needed to deploya virtual machine configuration and increases the costs associatedtherewith. For example, the virtual machines are only deployed when asystem administrator is present and after the system administratorperforms the manual customization to the configuration.

FIG. 5 presents an illustration of a prior art grid management system510 that copies static software images 520-540 to configure sets ofallocated resources on multiple hardware nodes 550-570. In this figure,the grid management system 510 retrieves a static image 520 from animage store 580 and applies the image 520 as retrieved to an allocatedset of resources for a virtual machine on node 570. To customize theconfiguration according to a particular user's specifications, a systemoperator manually customizes the configuration per the user'sspecifications.

FIG. 5 illustrates still another shortcoming of current virtualizationsolutions and the grid management system 510 utilized by some hostingservice providers. In this figure, the grid management system 510becomes a bottleneck within the hosting service system. The gridmanagement system 510 is the central point through which all virtualmachines on all nodes are configured. As such, each image forconfiguring each virtual machine first passes through the gridmanagement system 510 before reaching the corresponding node. Thiscauses the nodes to compete for the input/output resources of the gridmanagement system 510. For large configurations or for periods of hightraffic volume, the effectiveness of the grid is compromised as delaysarise during the configuration of a virtual machine.

Accordingly, there is a need to provide a more robust, automated,efficient, and synergized virtualization solution for multi-node hostingservices. Such a virtualization solution should operate over multiplehardware nodes in order to optimally distribute the resources of themultiple nodes amongst multiple different virtual machine configurationsirrespective of the hypervisors operating on each node. There is furthera need for such a virtualization solution to automatedly customconfigure partitioned resources without manual user intervention basedon user-specified parameters for a virtual machine configurationirrespective of the quality or quantity of user customization.

SUMMARY OF THE INVENTION

Some embodiments provide a hosting system and method for automatedlydistributing, configuring, and managing virtual machines across a groupof hardware nodes where at least two nodes in the group of nodes arecontrolled by different hypervisors. The different hypervisors includehypervisors of different types (e.g., type 1, type 2, etc.) and/ordifferent vendor implementations. In some embodiments, the automatedhosting system includes a (1) hypervisor management module and/or a (2)utility management module to implement the hosting functionality.

The hypervisor management module of some embodiments manages the nodesand the various virtual machines operating on the nodes with thedifferent hypervisors. The hypervisor management module normalizes themessaging for each of the distinct hypervisors so that there is auniform set of control messages and provisioning messages to control theallocation and operation of the virtual machines operating across thenodes irrespective of the hypervisors operating on the nodes. Thehypervisor management module uses various application programminginterfaces (APIs) to convert the normalized set of control messages andprovisioning messages to the particular control messages andprovisioning messages that are specific to the hypervisor of each targetnode. In this manner, the hypervisor management module provides amulti-tenancy solution whereby each node and the hypervisor of the nodeis a tenant managed by the hypervisor management module.

In the multi-tenancy hosting system of some embodiments, the hypervisormanagement module stores a mapping that identifies the location of eachvirtual machine to the one or more nodes hosting the virtual machine andthe corresponding hypervisors operating on each such node. Thehypervisor management module then formats the messages so that they areexecutable by the hypervisor at the node. The hypervisor managementmodule issues provisioning messages to the various hypervisors in orderto allocate resources needed to deploy a virtual machine configurationat one or more nodes. For example, the hypervisor management moduleissues messages to create, delete, or copy virtual machines to and fromthe nodes. The hypervisor management module issues control messages tothe various hypervisors in order to start, stop, resume, reboot, take asnapshot, restore to a snapshot, report statistics, save an image, ormigrate any virtual machine at any node irrespective of the hypervisoroperating at the node.

The hypervisor management module of some embodiments generates themapping of the virtual machines across the nodes and a mapping of theused and unused resources for the entire grid of nodes by monitoring theoperation of the virtual machines, hypervisors, and nodes within thegrid of nodes. The hypervisor management module provides suchfunctionality by interfacing with two or more distinct virtualizationhypervisors operating on two or more hardware nodes. Through theinterface, the hypervisor management module is able to determine theavailability of resources (e.g., CPU cycles, available memory, blockinput/output, network traffic, etc.) on each node in the group of nodes.

In some embodiments, the hypervisor management module retrieves the datafrom the hypervisors operating on the nodes. In other embodiments, thehypervisor management module bypasses the virtualization hypervisorsoperating on the nodes and directly retrieves usage and availabilitydata from the hardware resources and/or virtual machines of each nodedirectly. In still other embodiments, the hypervisor management moduleretrieves some usage data from the hypervisors while retrieving otherdata directly from the hardware resources of the nodes. The data isnormalized to create the unified view of the resource usage and theavailability of all hosting resources for all nodes in the group ofnodes.

By storing and combining the resources from each node, the hypervisormanagement module of some embodiments creates a unified view for theused and unused resources of the group of nodes. In this manner, thehypervisor management module seamlessly merges resources from physicallyseparate hardware nodes that operate using incompatible hypervisors tocreate a single logical amalgamation of resources irrespective of thehypervisor type or vendor implementation. Scheduling and deploymentdecisions for additional virtual machines are based on this unifiedview.

The hypervisor management module of some embodiments utilizes theunified view of the group of nodes within the grid to facilitate theefficient deployment of virtual machines across the nodes. The efficientdeployment of virtual machines across the nodes includes identifying andallocating resources of (1) underutilized nodes or (2) nodes that aremore optimal to host a virtual machine relative to another node based onresources of the node (e.g., hardware resources and hypervisorefficiency). Accordingly, the hypervisor management module controls andmanages this efficient deployment of the virtual machines across thegroup of nodes without user intervention in order to fully maximize theusage of the resources while optimizing the performance of each virtualmachine based on the performance advantages of each hypervisor operatingacross the nodes and the available resources of the nodes.

In some embodiments, the hypervisor management module, using the variousmessages, tasks the utility management modules of the correspondingnodes to custom configure the allocated resources once the hardwareresources have been allocated. The utility management modules customconfigure the allocated resources based on a user's specified virtualmachine configuration. In some embodiments, the hypervisor managementmodule instantiates each of the utility management modules. In some suchembodiments, the hypervisor management module instantiates a utilitymanagement module by directing the utility management modules to mount aremote file system containing one or more scripts. The hypervisormanagement module then identifies one or more of the scripts to beexecuted locally by the utility management module on a particular nodeat which the utility management module operates. These scripts directthe utility management module to perform specific tasks on theparticular node that assist in the automated configuration of one ormore user virtual machines on the particular node as each utilitymanagement module is responsible for configuring the hardware resourcesallocated for a particular virtual machine configuration on a particularnode.

To configure the resources, the utility management module identifies (1)an operating system (OS) image from an OS image store and (2)application images from an application image store that collectivelyprovide the software resources specified within the user virtual machineconfiguration. The utility management module of some embodimentsretrieves compressed images of the various OS and software applicationsfrom the image stores. Since each utility management module retrievesthe images required for the virtual machines on its corresponding node,some embodiments remove the bottleneck associated with a central imageretrieval solution that retrieves images for all nodes of the grid ofnodes. The utility management module then decompresses the retrievedimages locally on the node. In this manner, the utility managementmodule only transfers data from the image stores that is needed for aparticular virtual machine configuration while preserving blockinput/output (e.g., input/output bandwidth). In some embodiments, theutility management module modifies the retrieved images based on thespecified parameters within the virtual machine configuration. Theutility management module directly accesses the file system allocated tothe virtual machine and applies (e.g., partitions, formats, installs,etc.) the customized images to the provisioned resources of the virtualmachine configuration in order to instantiate or modify a virtualmachine configuration.

The utility management module of some embodiments securely deletesvirtual machine configurations operating on a particular node byaccessing the file system allocated for the virtual machineconfiguration and securely deleting the data from the file system. Someembodiments securely delete the data by writing random data to the filesystem. Some embodiment securely delete the data by zeroing out allstorage bits of the disk storage previously allocated for use by thevirtual machine.

The utility management module of some embodiments encrypts the virtualmachine configuration. Such encryption provides an added layer ofsecurity to prevent a configuration error or other software bug frombeing exploited by one virtual machine that attempts to access the blockdevice or file systems of other virtual machines operating on the sameor other nodes. Additionally, the utility management module facilitatesthe migration of virtual machines from one node to another node wheninstructed by the hypervisor management module. In such instances, theutility management module of a first node passes virtual machineconfiguration data directly to a second node within the group of nodessuch that the virtual machine configuration may be instantiated on thesecond node. The configuration is then deleted from the first node bythe utility management module.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a single computing device running virtualization inorder to function as two or more different computing devices with eachdevice having distinct sets of hardware and software resources.

FIG. 2 illustrates a type 1 hypervisor of a prior art virtualizationimplementation.

FIG. 3 illustrates a type 2 hypervisor of a prior art virtualizationimplementation.

FIG. 4 illustrates a prior art virtualization solution implementing aproprietary hypervisor manager.

FIG. 5 presents an illustration of a prior art grid management systemthat copies static software images to configure sets of allocatedresources on multiple hardware nodes.

FIG. 6 illustrates a hypervisor management module that controls theoperation of the virtual machines across the grid of nodes and monitorsthe resources of the nodes in accordance with some embodiments.

FIG. 7 illustrates some operations performed by the hypervisormanagement module for allocating a set of resources of a node from agroup of nodes based on a specified virtual machine configuration inaccordance with some embodiments.

FIG. 8 illustrates some operations performed by a utility managementmodule of a particular node to custom configure a set of resources thatthe particular node has allocated for a virtual machine configuration.

FIG. 9 illustrates a hosting system that implements some embodiments ofthe invention.

FIG. 10 presents a more detailed illustration of the components of thehypervisor management module and their interactions with the hypervisorsand utility management modules of the nodes.

FIG. 11 illustrates a control message for starting the operation of avirtual machine passed from the RMS to the hypervisor management modulein accordance with some embodiments.

FIG. 12 illustrates a provisioning message for adding a component to anexisting virtual machine operating on a particular node in the grid ofnodes in accordance with some embodiments.

FIG. 13 presents a process for collecting statistical data from the gridof nodes in accordance with some embodiments of the hypervisormanagement module.

FIG. 14 presents a process performed by a hypervisor management moduleof some embodiments for the allocation of resources for a newlyspecified virtual machine configuration.

FIG. 15 presents a process implemented by the hypervisor managementmodule of some embodiments to optimally allocate resources from the gridof nodes to a virtual machine configuration.

FIG. 16 presents a process performed by the hypervisor management modulefor modifying an existing virtual machine configuration for a virtualmachine that operates within the grid of nodes.

FIG. 17 presents a grid of sixteen nodes with three nodes expanded toillustrate the utility management module and various virtual machinesrunning on each expanded node.

FIG. 18 illustrates various functional components of a utilitymanagement module for configuring resources allocated for a virtualmachine configuration in accordance with some embodiments of theinvention.

FIG. 19 presents a process performed by a utility management module toconfigure an allocated set of resources in accordance with someembodiments.

FIGS. 20A-20E provide a utility management module of some embodimentsand some operations performed by the utility management module toconfigure a virtual machine according to a user-specified configuration.

FIG. 21 conceptually illustrates the ability of a utility managementmodule of some embodiments to access file systems of other virtualmachines operating on the same node as the utility management module.

FIG. 22 illustrates the various customizations performed by a utilitymanagement module in accordance with some embodiments of the invention.

FIG. 23 presents a process implemented by a utility management module ofsome embodiments to customize an image in accordance with FIG. 22.

FIG. 24 illustrates an exemplary set of operations performed by autility management module of some embodiments to modify a configuration.

FIG. 25 conceptually illustrates thresholds for various resources of arunning virtual machine that are used by a utility management module ofsome embodiments to automatically scale the virtual machine resources.

FIG. 26 presents a process performed by a utility management module ofsome embodiments to automatically scale a user virtual machine when oneor more resource thresholds for the virtual machine have been met.

FIG. 27 conceptually illustrates an example of optimizing a virtualmachine installation and configuration in accordance with someembodiments of the embodiment.

FIG. 28 presents a process for securely deleting a virtual machineautomatically using a utility management module of some embodiments.

FIG. 29 illustrates an interfacing between utility management modules inorder to migrate a virtual machine configuration from one node toanother.

FIG. 30 presents a process for migrating a virtual machine from a firstnode to a second node in a grid of nodes in accordance with someembodiments.

FIG. 31 illustrates a computer system with which some embodiments of theinvention are implemented.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

I. Overview

Some embodiments provide a hosting system and method for automatedlydistributing, configuring, and managing virtual machines across a groupof hardware nodes. In some embodiments, the virtual machines includesystem virtual machines. A system virtual machine is a particularconfiguration of a computer system platform with its own file system andoperating system (OS). The computer system platform may also includevarious application programs that are installed on the file system andthat are compiled to directly execute on the OS platform. A systemvirtual machine is therefore a complete configuration that isindependently operable on a hardware node.

On a particular hardware node of the hosting system that hosts multiplevirtual machines, a system virtual machine operates independent of otheroperating systems of the other virtual machines operating on theparticular hardware node. In this manner, multiple OS environments(e.g., Microsoft Vista®, Linux, etc.) co-exist on a node withoutmutually interfering with each other's operations. In other words, ahardware node of some embodiments concurrently executes virtual machinesof different computer system configurations for one or more users usinga single set of hardware resources of the node.

System virtual machines are different than process virtual machines.Unlike system virtual machines, a process virtual machine does notinclude its own OS. Rather, a process virtual machine runs as anapplication on a host operating system. The process virtual machineprovides a platform independent programming environment for applicationprograms that interface with the process virtual machine in order to runon the host operating system. The application programs therefore cannotrun on the host operating system without the process virtual machine.Similarly, the process virtual machine cannot run the applicationprograms without the host operating system.

In some embodiments, a node is a particular physical machine such as aserver or other computer device/machine that includes processingresources, memory resources, block/storage resources, network resources,etc. that implement and execute the one or more computer systemconfigurations of the virtual machines. The node also includes ahypervisor that operates at a lower layer than the virtual machines onthe node.

The hypervisor manages the access of the virtual machines on a node tothe underlying hardware resources of the node. The hypervisor managesthe hardware resources of a particular node by partitioning a set of thehardware resources to each virtual machine operating on the particularnode. In some embodiments, the hypervisor provides exclusive use of apartitioned set of the hardware resources to a particular virtualmachine. In this manner, some embodiments guarantee that an amount ofhardware resources is always available to each virtual machine. Thevirtualization or mapping of the hardware resources occurs transparentlyto the operations of the virtual machine causing each virtual machine tobelieve that they are operating using a dedicated set of resources. Suchtransparent virtualization or mapping of resources prevents one virtualmachine from accessing or interfering with resources of another virtualmachine. For example, a virtual machine is prevented from accessing thefile system or memory (i.e., random access memory) of another virtualmachine on the same node.

In some embodiments, within the group of nodes for hosting the virtualmachines, the hardware resources of at least two nodes are controlled bydifferent hypervisors, a first hypervisor operating on a first node anda second different hypervisor operating on a second node. In someembodiments, the different hypervisors include hypervisors of differenttypes. For instance, a type 1 hypervisor controls the resources of thefirst node and a type 2 hypervisor controls the resources of the secondnode.

In some embodiments, the different hypervisors include hypervisors ofdifferent vendors. For instance, different vendors may implement thesame type 1 hypervisor but with different messaging interfaces where thedifferent messaging interfaces define two or more messaging protocols,commands, and interfaces to the operating systems being virtualized bythe hypervisors. Additionally, the same hypervisor vendor may havemultiple different hypervisor implementations. For example, the samevendor implements different hypervisors that support virtualization ofdifferent sets of guest operating systems. Currently, no singlehypervisor currently provides an all-in-one virtualization solution thatvirtualizes every commercially available operating system (e.g., a firsthypervisor that supports virtualization of Windows® and MAC OS® and asecond hypervisor that supports virtualization of LINUX and Solaris®).

In some embodiments, the different hypervisors include hypervisors thatsupport virtualization of a particular set of hardware. For example, afirst hypervisor supports virtualization of an x86 based processor andnot a RISC based processor, whereas a second hypervisor supportvirtualization of the RISC based processor and not the x86 basedprocessor.

In some embodiments, the different hypervisor provide different levelsof virtualization. For instance, some hypervisor provide fullvirtualization, partial virtualization, paravirtualization, etc.

To provide an automated hosting system for such a hosting environmentwith different hypervisors, some embodiments provide (1) a hypervisormanagement module and/or (2) one or more utility management modules toimplement the automated hosting functionality. The hypervisor managementmodule of some embodiments manages the nodes and the various virtualmachines operating on the nodes with the different hypervisors. Thehypervisor management module normalizes the messaging for each of thedistinct hypervisor types and distinct hypervisor vendorimplementations. This allows the hypervisor management module of someembodiments to provide a uniform set of control and provisioningmessages for controlling the allocation and operation of the virtualmachines operating across the nodes irrespective of the hypervisoroperating on the node. The hypervisor management module converts theuniform set of control and provisioning messages to differentprovisioning messages and control messages that are specific to thehypervisor of each target node. Consequently, the hypervisor managementmodule provides a multi-tenancy solution for the different hypervisorsoperating across the group of nodes where each node and thecorresponding hypervisor operating at the node is a tenant.

The hypervisor management module therefore enables a scalable hostingsystem whereby new or different hypervisors may be integrated into oneor more nodes of the hosting system at any time. To support the new ordifferent hypervisors, the hypervisor management module of someembodiments requires only minimal updates to normalize the messaging fornew or different hypervisors that are introduced into the hostingsystem.

The hypervisor management module uses the provisioning messages tooptimally allocate the various resources needed for deploying a virtualmachine configuration from one or more nodes that are best suited tohost the configuration. For example, the hypervisor management moduleoptimizes performance of a virtual machine based on the hypervisor(s)operating on the nodes and the hardware resources of the nodes wherecertain hypervisors more efficiently execute networking messageexchanges and other hypervisors more efficiently execute localinput/output (I/O) message exchanges. The provisioning messages includemessages to create, delete, scale resources, or copy a virtual machine.These provisioning messages are described in further detail in SectionIII B below. It should be apparent to one of ordinary skill in the artthat the list of provisioning messages is not meant to be a completelist of all provisioning messages but rather an exemplary set ofmessages used by the hypervisor management module of some embodiments.

The hypervisor management module uses the control messages to controlthe operation of any virtual machine deployed across the nodesirrespective of the different hypervisors managing the resources of eachnode. Some examples of control messages include messages to start, stop,resume, reboot, take a snapshot, restore to a snapshot, reportstatistics, save an image, or migrate operation of a virtual machine.These control messages are described in further detail in Section III Bbelow. It should be apparent to one of ordinary skill in the art thatthe list of control messages is not meant to be a complete list of allcontrol messages but rather an exemplary set of messages used by thehypervisor management module of some embodiments. Additionally, thehypervisor management module monitors the virtual machine across thenodes to collects statistics about the hardware resources, hypervisor,or virtual machines of a node.

In some embodiments, the hypervisor management module leveragesfunctionality provided by the utility management modules, also referredto as utility virtual machines (depicted in the figures as “UVM”). Insome embodiments, the utility management modules are virtual machinesthat locally reside on each node in the group of nodes. The utilitymanagement modules provide utility functions that assist the hypervisormanagement module in automatically installing, configuring, and deletingvirtual machines from the group of nodes. In some embodiments, theutility functionality includes (1) automatedly instantiating customvirtual machine images onto the utility management module'scorresponding node based on user-specified parameters within aparticular virtual machine configuration, (2) automatedly modifyingexisting configurations by adding to or removing from the softwarecomponents of the existing configuration, (3) securely deleting virtualmachines, and (4) encrypting the virtual machines.

FIG. 6 illustrates a hypervisor management module 610 that controls theoperation of the virtual machines across the grid of nodes and monitorsthe resources of the nodes in accordance with some embodiments. Thecomponents of the hypervisor management module 610 illustrated in FIG. 6and other components are described in further detail with reference toFIG. 10 below. The hypervisor management module 610 interfaces with atleast three different nodes 620, 625, and 630. In this figure, the nodes620, 625, and 630 include different resources with different hypervisorsvirtualizing the resources of the node. For instance, the resources ofnode 620 are virtualized using a first hypervisor (e.g., Hypervisor_1 ofFIG. 6) and the resources of nodes 625 and 630 are virtualized using asecond hypervisor (e.g., Hypervisor_2 of FIG. 6), where the first andsecond hypervisors are different. As noted above, the differenthypervisors include type 1, type 2, and other types of hypervisors. Thedifferent hypervisors may also include hypervisors of the same type thatare implemented differently by different vendors.

To control the operations of the nodes 620, 625, and 630, the hypervisormanagement module 610 interfaces with the various hypervisors of thenodes. The hypervisor management module 610 interfaces with thehypervisors by adapting control messaging such that the control messagesare executable on each node irrespective of the hypervisor operating onthe node.

Specifically, a virtual machine controller 635 of the hypervisormanagement module 610 generates the control message. Then, using one ormore different proxy managers 645, the hypervisor management moduleinterfaces the control message to a particular hypervisor of a node. Inthis manner, the hypervisor management module 610 of some embodimentsnormalizes the distinct interfaces of each of the different hypervisorsin order to provide a multi-tenancy solution. It should be apparent toone of ordinary skill in the art that the hypervisor management moduleof some embodiments is scalable to support any number of hypervisors viaa corresponding proxy manager.

The proxy managers 645 include application programming interfaces (APIs)that specify the protocols and functions for communicating with each ofthe different hypervisors on the nodes 620, 625, and 630. For example,for node 620 that is virtualized with a first hypervisor, the hypervisormanagement module 610 interfaces with this node through a first proxymanager 640, and for nodes 625 and 630 virtualized with a secondhypervisor, the hypervisor management module 610 interfaces with node630 through a second proxy manager 650.

The proxy managers 640 and 650 translate the messages originating fromthe virtual machine controller 635 to messages that are executable bythe various hypervisors operating on each corresponding node. Forinstance, the hypervisor management module 610 using virtual machinecontroller 635 in conjunction with the appropriate proxy manager, (1)configures, translates, and sends a first message 660 to cause a firstvirtual machine on node 620 to commence its operations, (2) configures,translates, and sends a second message 670 to cause a second virtualmachine on the node 625 to halt its operations, and (3) configures,translates, and sends a third message 680 to cause a first virtualmachine on the node 630 to halt its operations.

Also, through the interfaces to the nodes 620, 625, and 630, thehypervisor management module 610 monitors resource statistics of eachnode, such as available CPU processing resources, memory, disk storage,block input/output, or network traffic passing into and out of a virtualmachine. These statistics may be used to indicate the availability ofthe various resources on each node. In some embodiments, the monitoringis performed using a status poller 695 of the hypervisor managementmodule 610, where the status poller 695 retrieves the statistics onparticular intervals, on particular triggers, or on a continuous basisin order to create a real-time or near real-time view of the usage andavailability of resources across the grid of nodes.

In some embodiments, the hypervisor management module 610 retrieves thestatistics 690 from the node directly (e.g., directly from the hardwareresources). For instance, through hardware probes and low-levelprocesses, the hypervisor management module is able to retrievestatistics directly from the hardware. Additionally, the hypervisormanagement module 610 is able to retrieve the statistics 690 byinterfacing with the virtual machines either directly or indirectlythrough a utility management module described below. Further still, thehypervisor management module 610 of some other embodiments retrieves thestatistics 690 from the hypervisor operating on each node where eachhypervisor may keep a copy of the statistics within a local database. Itshould be apparent to one of ordinary skill in the art that thehypervisor management module of some embodiments is able to retrievestatistics from some or all such sources in addition to various othersources.

In this manner, the hypervisor management module 610 seamlessly controlsand merges resources from physically separate hardware nodes to create asingle logical amalgamation of resources. In so doing, the hypervisormanagement module 610 seamlessly gathers, normalizes, and mergesstatistics from physically separate hardware nodes. As a result, asystem administrator no longer has to view the group of nodes asseparate resources, but rather can interface with the group of nodes asif they were a single resource irrespective of the various hypervisorsthat operate across the grid of nodes or the various hardware resourcespresent on each node in the grid of nodes. This unified view of thegroup of nodes facilitates the efficient deployment of virtual machinesacross the nodes such that the utilization of the resources of all nodesis maximized.

FIG. 7 illustrates some operations performed by the hypervisormanagement module 720 in order to allocate a set of resources of a nodefrom a group of nodes based on a user specified virtual machineconfiguration in accordance with some embodiments. Specifically, thehypervisor management module 720 interfaces with a first hypervisor 740of node 730 and a second different hypervisor 745 of node 780.

As noted above with reference to FIG. 6, the hypervisor managementmodule 720 is able to develop a mapping of the available and usedresources across the grid of nodes by monitoring the statistics of thenodes 730 and 780. From this mapping, the hypervisor management module720 identifies node 730 as a node with sufficient resources to host thevirtual machine configuration 710 based on requirements specified withinthe virtual machine configuration 710. Additionally, in someembodiments, the hypervisor management module 720 identifies node 730 asan optimal node for hosting the virtual machine configuration 710. Insome embodiments, such a determination is based on the efficiency withwhich the hypervisor operating on the node executes the virtual machineconfiguration 710 or is based on the various hardware resources on thenode 730. It should be apparent to one of ordinary skill in the art thatvarious other criteria may be used to determine the optimal node forhosting configuration 710.

After identifying the node 730 with the sufficient resources forhosting, the hypervisor management module 720 requests that thehypervisor 740 provision a specified set of resources 750 for theoperation of the virtual machine configuration 710. In response, thehypervisor allocates the set of resources 750 to the virtual machineconfiguration 710 and restricts other virtual machine configurationsfrom accessing the provisioned set of resources 750.

The hypervisor management module 720 then passes the virtual machineconfiguration 710 to the utility management module 760 for the utilitymanagement module 760 to configure the allocated resources 750. FIG. 8illustrates some operations performed by a utility management module 820operating on a particular node 830 to custom configure a set ofresources 850 of the particular node 830 that have been allocated by thehypervisor management module 805 for a virtual machine configuration815.

In some embodiments, the utility management module 820 is a softwaremodule (e.g., virtual machine) operating within each node that hasinput/output interfaces for communicating with the hypervisor managementmodule 805, the hypervisor 860 of the particular node 830, the hardwareresources 850, and the image store 825. The hypervisor management module805 instantiates each utility management module as needed. For instance,when adding a new virtual machine to a particular node, the hypervisormanagement module 805 provisions the set of resources 850 for thevirtual machine and then directs the utility management module 820 toconfigure the resources. The utility management module 820 beginsconfiguring the allocated set of resources 850 after receiving a virtualmachine configuration 815 from the hypervisor management module 805. Insome embodiments, the virtual machine configuration 815 specifies a setof scripts on a remote file system that the utility management module820 locally executes at the particular node in order to configure theallocated resources.

To configure the allocated resources, the utility management module 820partitions and formats the allocated resources to create a file systemfor the virtual machine. In some embodiments, the utility managementmodule 820 also partitions and formats a local logical file system. Theutility management module 820 builds the user-specified virtual machineon the local file system. When the configuration is complete, theutility management module 820 then copies the virtual machine from thelocal file system to the file system of the allocated set of resources.Once the virtual machine resides on the allocated resources, the virtualmachine may commence operations.

It should be apparent to one of ordinary skill in the art that theutility management module executes each of the partitioning, formatting,and custom configuring operations based on different scripts stored onthe remote file system. In some embodiments, the hypervisor managementmodule controls each instance of these operations for each utilitymanagement module. Other operations performed by the utility managementmodule may similarly reside on the remote file system where eachinstance of the utility management module executing one or more suchscripts is controlled by the hypervisor management module. For example,the remote file system may include scripts for retrieving images,modifying images, installing images, etc. as described below. In someembodiments, each utility management module mounts the remote filesystem in order to access the scripts. However, it should be apparent toone of ordinary skill in the art that in some embodiments the scriptsare locally stored within the file system of each utility managementmodule.

In some embodiments, the file system of the virtual machine is createdto store all configuration files for the virtual machine operatingsystem, application programs, and user data. The hypervisor managementmodule in conjunction with either the hypervisor of a node or theutility management module of a node creates the file system from anarray of fixed-sized blocks of the block device (e.g., disk storagedevice) of the node. The blocks may be clustered into sectors. The filesystem then organizes the sectors to create, store, move, and delete thevarious files and directories of a virtual machine. An operating systemcontrols access to the file system. However, a hypervisor may introducea layer of abstraction between the actual sectors of the block deviceand the file system managed by an operating system of a virtual machine.In such configurations, the operating system believes that it is inexclusive control over the file system and thus in control of theunderlying block device. Yet, actual control over the physical blockdevice is retained by the hypervisor. The file system thus represents aportion of a block device of a node that is set aside to store data andconfiguration files of a virtual machine, where the block device itselfis shared amongst multiple different virtual machines.

In some embodiments, a file system is hierarchically organized. Forexample, a hierarchical organization of data may include a nesteddirectory structure with each directory storing one or more files orother directories. Some embodiments leverage such a hierarchicalorganization and directory structure when building the virtual machines.This allows the utility management module 820 to build the virtualmachine modularly by separating, for example, operating system filesfrom application program files and by modularly installing drivers for aparticular operating system into a nested directory of the particularoperating system. It should be apparent to one of ordinary skill in theart that a flat structure may also be used to organize the file system.In some embodiments, the hypervisor retains the structure of all filesystems of all virtual machines operating on a node in order to permitor restrict access by the virtual machines and the utility managementmodule.

To build the user-specific virtual machine, the utility managementmodule 820 retrieves a software image 810 from an image store 825 andtemporarily copies the image to the local file system of the utilitymanagement module 820. In some embodiments, the image store 825 includesa separate image store for storing operating system images and aseparate image store for storing application images that operate withone or more operating systems.

In some embodiments, the utility management module 820 retrieves acompressed image from the image store 825 whereby only actual data andnot redundant or empty blocks of an image is transferred between theimage store 825 and the utility management module 820. The utilitymanagement module 820 decompresses the data and modifies the retrievedimage 810 based on parameters specified within the received virtualmachine configuration 815 before using the modified image to configurethe user virtual machine.

In some embodiments, modification of the retrieved image 810 includesmodifying default parameters within the configuration image 810 such asspecifying IP address values for firewall settings or specifying aunique hostname for the operating system of the virtual machineconfiguration. In some embodiments, modification of the retrieved image810 includes removing extraneous components within the retrieved image810 that are not needed for the virtual machine configuration. In thismanner, the utility management module 820 minimizes the file size of theimage to be copied to the allocated disk storage. This allows for theutility management module 820 to bring online the virtual machineconfiguration faster as less data has to be copied and resources, suchas disk space and I/O bandwidth, are not wasted on copying extraneousdata elements not relevant to tasks performed by the virtual machine.

In still other embodiments, modification of the retrieved image 810includes adding components that are not within the default image 810retrieved from the image store. These software components may includeproprietary software applications particular to a user. Since theutility management module 820 allows for the customization of thesoftware configuration for the resources, any such user configurationcan be applied automatically without manual intervention.

As noted above, once the virtual machine is completely configured withinthe local file system of the utility management module 820, the utilitymanagement module 820 then copies the virtual machine over to the filesystem created for the virtual machine from the allocated set ofresources. In some embodiments, the utility management module directlyaccesses the file system in order to copy the modified virtual machineimages to the file system.

The utility management module of some embodiments directly accesses thefile system through the hypervisor of a node. In some such embodiments,the hypervisor is modified to permit specialized calls from the utilitymanagement module to access and modify the file systems of the othervirtual machines on the same node.

The utility management module of some embodiments encrypts the operatingconfiguration for a virtual machine so that other virtual machines onthe same node or other nodes will be unable to access components of thevirtual machine. The utility management module receives an encryptionkey as part of the virtual machine configuration. After installing theconfiguration, the utility management module then uses the key toencrypt the operating configuration. Subsequent access to theconfiguration of the virtual machine then requires the use of thespecific encryption that only select personnel have (e.g., systemadministrator). In some embodiments, encrypting the operatingconfiguration includes encrypting part of the virtual machine filesystem that stores the essential configuration data or user data. Inother embodiments, encrypting the operating configuration includesencrypting the entire file system of a virtual machine.

The utility management module of some embodiments provides furthersecurity functionality by securely removing a software configurationfrom an allocated set of resources. In some such embodiments, theutility management module uses the access rights to the allocatedresources (e.g., disk storage of the node that has been allocated to thevirtual machine) in order to write random data to the disk storagebefore reallocating the disk storage to a different virtual machine. Inthis manner, some embodiments avoid the issue concerning prior artimplementations that simply reallocate the disk storage without deletingthe data first, allowing subsequent users the ability to extractpotentially sensitive and confidential data from the prior virtualmachine configuration.

In some embodiments, the utility management module of some embodimentsoperates with the hypervisor management module to migrate a virtualmachine operating from one node to another. For instance, beforeperforming maintenance on a first node, the hypervisor management moduleinstructs the utility management module on the first node to migrate thevirtual machines from the first node to a second node. The utilitymanagement module of the first node then interfaces with a utilitymanagement module of the second node and the virtual machines operatingon the first node are transferred to the second node. The virtualmachines may then begin operations on the second node as they are haltedon the first node and maintenance is performed on the first node. Inthis manner, system administrators may automatically take down nodes formaintenance without resulting in disruption or downtime to any virtualmachines previously operating on those nodes.

Several more detailed embodiments of the invention are described in thesections below. Before describing these embodiments further, Section IIprovides a conceptual architectural diagram for the various modules forimplementing the automated hosting environment of some embodiments.Next, Section III describes some embodiments of a hypervisor managementmodule for provisioning a set of resources to a virtual machineconfiguration and for managing the provisioned set of resources. SectionIV describes some embodiments of a utility management module for customconfiguring a provisioned set of resources to a virtual machineconfiguration. Lastly, Section V describes a computer system whichimplements some of the embodiments of the invention.

II. Architecture

FIG. 9 illustrates a hosting system 900 and the various components ofthe hosting system 900 that implement some embodiments of the invention.This system automatedly receives new or modified server configurationsthrough front-end user interface (UI) logic and then automatedly deploysthe server configuration onto a grid of hardware nodes through back-endplacement logic. In some embodiments, the hosting system 900 provideshosting services for multiple unrelated users over the shared grid ofhardware nodes. As shown in FIG. 9, the hosting system 900 includes: (1)a service request server 910, (2) front-end provisioning manager 920,(3) a resource management system module 930, (4) a hypervisor managementmodule 940, (5) a data storage 950, (6) a statistics storage 955, (7) anoperating system image store 960, (8) an application image store 963,and (9) a grid of hardware nodes 965.

The service request server 910 (1) receives communications (e.g.,service requests) from external users through a network 915 and (2)routes the communications to the front-end provisioning manager 920. Insome embodiments, the service request server 910 is a web server, whichcommunicates to a user through a network 915 such as the internet. Insuch embodiments, a user accesses the hosting system 900 through theuser's web browser or through a downloadable client application 905which may reside on the user's desktop computer, portable notebookcomputer, personal digital assistant (PDA), digital cellular telephone,or other electronic communication device. In other embodiments, the usercan access the hosting system 900 through communication networks otherthan the internet. For instance, the network 915 may include wirelessdata services (e.g., GPRS or other variations of 2G or 3G packet dataservices) or other electronic communication interfaces. In this manner,users may access the hosting system 900 while being located anywherethroughout the world.

The service request server 910 routes user communications to thefront-end provisioning manager 920. On an initial communication, thefront-end provisioning manager 920 passes the user communication to aregistration module (not shown) for user verification and authentication(e.g., username and password verification). In some embodiments, theregistration module is a fully automated component of the hosting system900 that performs the verification and authentication operations withouthuman intervention.

If the user is not an existing customer, the registration module of someembodiments presents a graphical interface with editable fields throughwhich the user enters additional identification information for creatinga user account. The user-specified information is then stored withindata storage 950 for subsequent authentication and authorization of theuser. If the user is an existing customer, the user's prior virtualmachine configurations and usage information are retrieved from the datastorage (i.e., database) 950. The information is passed to the front-endprovisioning manager 920.

The front-end provisioning manager 920 generates a graphical interfacethrough which users specify graphical representations for the variouscomponents of the virtual machine configurations hosted by the group ofresource nodes within the grid 965. In some embodiments, the graphicalrepresentations include sets of selected graphical items (e.g., icons),where each item represents a component of the server configuration. Forinstance, a user desiring to create a server configuration having a loadbalancer, multiple web servers, and a database server simply selects thethree graphical elements within the GUI that represent such components.In some embodiments, such selection occurs when users click on (e.g.,left-click, right-click, double-click, etc.) items within the graphicaldisplay to add, modify, or delete the graphical representations for suchitems, while some other embodiments allow users the ability to drag anddrop the graphical representations for such items across the graphicaldisplay. In some embodiments, each graphical representation includes oneor more configurable parameters associated with configuring resources orcharacteristics of a physical device in the grid of nodes represented bythe graphical representation. For example, the user selects a graphicalrepresentation for a data storage component of a virtual machineconfiguration and the user then specifies an amount of disk storage thatis to be allocated to the data storage.

In some embodiments, the specified configuration is scalable to increaseor decrease allocated resources in response to demand through simplemodification of the graphical representation. Accordingly, the resourcesfor any previously deployed virtual machine configuration may bemodified by a user at any time by simply modifying the graphicalrepresentations for the configuration. To facilitate the scaling of aserver configuration, the front-end provisioning manager 920 acts as auser interface manager that provides a tiered hierarchicalrepresentation of the server configuration.

Some embodiments of the front-end manager 920 further permit users theability to specify custom configuration parameters for each component ofthe configuration or for the configuration as a whole. For instance, thefront-end manager 920 of some embodiments allows users the ability tospecify a desired software configuration (e.g., operating systems,anti-virus protection, anti-spam protection, applications, etc.) tooperate in conjunction with the specified hardware configuration. Inaddition to specifying the operating system and applications to includewithin the user configuration, some embodiments permit users the abilityto further specify configuration settings within the selected operatingsystem and applications. For instance, a user can enter networkaddresses for load balancers and firewalls or specify hostnames as someexamples.

After the graphical specification for the server configuration iscomplete, some embodiments of the front-end manager 920 automaticallyprovide the configuration to the hosting system's back-end logic, whichis formed by the resource management system module 930 and thehypervisor management module 940. In some embodiments, the resourcemanagement system module 930 of the back-end logic receives thespecified configuration from the front-end manager 920 and performs alogical assignment (i.e., identifies a mapping) of the components withinthe configuration to the grid of hardware nodes 965. In someembodiments, the logical assignment is stored within the data storage950 where it can be later accessed by other components of the hostingsystem 900. In some embodiments, the data storage 950 includes one ormore databases that reside on one or more physical devices.

In some embodiments, the hypervisor management module 940 receives thelogical assignment from the resource management system module 930 orfrom the data storage 950 after the resource management system module930 stores the assignment in the data storage 950. As described infurther detail in Section III below, the hypervisor management module940 then automatically deploys the logical assignment across one or moreof the physical hardware nodes 965.

To deploy a configuration, the hypervisor management module 940 performsa constraint optimization process by considering a variety of factorsthat determine on which nodes to deploy the logical assignment of theuser-specified configuration. In some embodiments, the factors used inthe constraint optimization process include: (1) the unallocatedresources of the hardware nodes, (2) the behavioral characteristics ofthe specified configuration (e.g., where one web server of aconfiguration experiences greater loads than a second web server), (3)the behavioral characteristics of the hardware nodes (e.g., performancecharacteristics of the hypervisor operating on each node), (4) theindividual characteristics associated with each user (e.g., thelikelihood that a user configuration will be subject to hacking orspamming), and (5) the internal specific parameters (e.g., softwarelicensing costs). By considering these and other factors, the hypervisormanagement module 940 performs intelligent resource allocation thatoptimizes the performance of the configuration over the set of hardwarenodes 965 and maximizes the number of users able to use the nodes 965.

Some of the statistical information for enumerating these variousfactors for the constraint optimization process are retrieved from thestatistics storage 955. The statistics storage 955 is populated by thehypervisor management module 940. Specifically, in some embodiments, thestatus poller referenced in FIG. 6 and further described in FIG. 10performs the statistics gathering for the hypervisor management module940.

In some embodiments, the hypervisor management module 940 pulls thestatistical data from multiple various sources on each node. Thesesources include (1) the hypervisors (e.g., 970) operating over the nodes965, (2) the hardware resources of the nodes 965, and (3) the utilitymanagement modules of the nodes 965. Because each of these sources mayprovide statistical information differently, the hypervisor managementmodule 940 normalizes the data and formats the data before storing thedata in the statistics storage 955.

The hypervisor management module 940 automatically allocates the logicalassignment across one or more of the physical hardware nodes 965 byinterfacing with the hypervisor 970 operating on one or more nodesallocated to host the configuration. The hypervisor 970 manages theallocation of resources at the individual node level whereas thehypervisor management module 940 of some embodiments manages theallocation of resources at the grid level for all nodes within the grid965. In some embodiments, the hypervisor management module 940virtualizes a single functional component of the configuration acrossmultiple nodes 965. For instance, if a database server of aconfiguration requires a large allocation of disk space, the hypervisormanagement module 940 may deploy this database server over two nodessuch that the disk space for the server is distributed across multiplenodes.

In summary, the hypervisor 970 of each node allows for a non-conflictingprovisioning of a node's resources to two or more virtual machines andthe hypervisor management module 940 allows several such nodes withmultiple virtual machines and different hypervisors to operateseamlessly together. Through this seamless operation of two or morenodes with different characteristics (e.g., different hypervisors ordifferent hardware resources), the hypervisor management module 940 isable to create the grid 965 that provides a larger potential pool ofavailable resources for hosting virtual machine configurations of anysize.

Also operating on each node is a utility management module 980 of someembodiments, also referred to as a utility virtual machine (“UVM”). Theutility management module 980 is a hosting system provider instantiatedvirtual machine that runs on each node. As such, the utility managementmodule 980 receives a partitioned set of hardware resources (e.g.,processing resources, memory, block/storage, etc.) of a node on which itoperates.

In some embodiments, the utility management module 980 includes anoperating system and set of application modules (described in detailwith reference to FIG. 18 below) or remotely stored scripts that producevarious physical transformations on the particular machine or node onwhich the utility management module 980 is located. Some such physicaltransformations include customizing the resources allocated to a uservirtual machine by the hypervisor management module 940. Specifically,the utility management module 980 creates the user virtual machine byconfiguring the allocated resources with a set of software that performthe operations of the user virtual machine. The user specifies the setof software through the front-end manager 920. As described in furtherdetail in Section IV below, the utility management module 980instantiates, optimizes, and customizes the configurations by retrievingimages from one or more image store databases 960 and 963 and modifyingthe retrieved image according to user specifications. In this manner,the utility management module 980 alters the block device (by changingbits on the physical sectors of the block device) allocated to the uservirtual machine. The result of such alterations is to make operational aspecific machine implementation that includes the user's virtual machinefrom an otherwise non-configured set of hardware resources.

In some embodiments, the image store database 960 is a database storingseveral operating system software images. In some embodiments, the imagestore database 963 stores software images for applications to run inconjunction with a virtual machine configuration. It should be apparentto one of ordinary skill in the art that the applications image store963 may store any commercially available software application formultiple different operating systems and other proprietary developedsoftware applications. In this manner, the utility management module 980permits each virtual machine configuration operating on a node of thegrid 965 to be automatically instantiated and unique based on userparameters specified through the graphical user interface of thefront-end logic 920.

Together, the resource management system module 930, the hypervisormanagement module 940, and each of the utility management modules 980provide an automated method and system to provision and uniquelyconfigure an array of hardware nodes 965 for multiple users without theneed for human intervention. Accordingly, some embodiments of theinvention provide a hosting system 900 that rapidly deploys andconfigures a server configuration after the configuration is specified.Such a hosting system 900 can provide such services at any time. Shoulda user desire to allocate more resources (e.g., hardware, software, orother resources) or modify parameters for already allocated resources,the user may do so by making modifications through the front-end userinterface (e.g., by altering the graphical representation of theconfiguration presented through the front-end user interface). The usercan add, delete, and/or modify existing configuration resources throughthe front-end user interface of some embodiments.

The modifications are then automatically passed from the front-end tothe back-end where the hypervisor management module optimallyre-provisions the modified configuration. Specifically, the hypervisormanagement module automatically provides a different allocation for theresources across the hardware nodes to implement the modifiedconfiguration and the utility management modules reconfigure theresources. This further provisioning, allocating, and configuring occursautomatedly and without human intervention.

As a result, users may dynamically allocate or reconfigure their set ofallocated hardware and software resources via the front-end interface atany time. Such control allows users the ability to actively manage theirown configurations without third-party or human intervention.Furthermore, such control allows users the ability to instantaneouslyand granularly adjust their configuration to account for actual orexpected changes in demand. In this manner, some embodiments of theinvention create a hosting environment that simulates the user having avirtually unlimited scalable local server configuration while alsoproviding the benefit of access to and control of the simulated localserver configuration irrespective of the user's location.

It should be apparent to one of ordinary skill in the art that the gridof hardware nodes 965 includes several distinct physical servers orclusters of servers located in a single server farm or distributedacross multiple server farms in multiple disparate locations.Accordingly, the grid of hardware nodes 965 represents a cloud ofcomputing resources shareable by multiple users. One of ordinary skillwill appreciate that servers in other embodiments encompass anystandalone computational element that can process requests it receives.

In some embodiments, the grid of hardware nodes 965 is uniformly used toimplement all components of a server configuration. However, someembodiments segregate various functionality across groups of nodes. Forinstance, in some embodiments, a first grouping or cluster of nodes isused to implement the load-balancing servers of a configuration and asecond grouping or cluster of nodes are used to implement other servercomponents (e.g., web servers, database servers, etc.) of theconfiguration. In some such embodiments, the load-balancing servers arededicated F5 load balancing server appliances that can be configured towork in conjunction with the other nodes of the grid.

In some embodiments, the grid of nodes contains an inter-communicationpathway by which each node shares data with other nodes of the array andthe hypervisor management module. Through, the inter-communicationpathway, physically separated nodes together operate as a singlefunctional unit.

Additionally, as mentioned above, the various physical resources of eachnode can be logically partitioned and allocated to one or more virtualmachines. For instance, each node in the grid of hardware nodes 965includes at least one processing unit, where through the variouspartitioning, allocation, and deployment performed by the hypervisormanagement module, hypervisor, and/or utility management module, eachphysical processing unit conceptually operates as multiple separateprocessing units for two or more virtual machines of the node. Otherresources of a node (e.g., memory, disk space, network bandwidth, etc.)can also be logically split to be shared by multiple users.

It should be apparent to one of ordinary skill in the art that thearchitecture depicted in FIGS. 9 and 10 do not encompass all embodimentsof the invention. Some embodiments of the architecture may include othervarious functional components to work in conjunction with or instead ofthe enumerated components illustrated in FIGS. 9 and 10. Additionally,some embodiments define the hypervisor management module as a separatefunctional device than each of the utility management modules on eachnode. However, it should be apparent to one of ordinary skill in the artthat in some embodiments the functionality of the utility managementmodules is integrated into the functionality of the hypervisormanagement module and the below described functionality for thehypervisor management module and the utility management modules isencompassed within a single device or functional entity. As such, thehypervisor management module of some embodiments performs the virtualmachine resource allocation and monitoring, but also performs thepartitioning, formatting, configuring, and modifying of the resources inorder to fully instantiate a virtual machine on a node.

III. Hypervisor Management Module

A. Hypervisor Management Module Architecture

FIG. 10 presents a more detailed illustration of the components of themulti-tenancy hypervisor management module of some embodiments and theirinteractions with the hypervisors, utility management modules, andhardware resources of the nodes. In this figure, the hypervisormanagement module 1010 includes: (1) an API message processor 1020, (2)a virtual machine deployment scheduler 1030, (3) a virtual machinecontroller 1040, (4) a data store 1050, (5) a status poller 1060, (6) anormalization engine 1070, and (7) hypervisor proxy managers 1075 and1080.

The API message processor 1020 receives and processes messages receivedfrom the resource management system described with reference to FIG. 9above. These messages may specify a logical assignment of a virtualmachine configuration that was specified or modified using the front-endprovisioning manager. The API message processor 1020 passes theprocessed messages to the deployment scheduler 1030.

The deployment scheduler 1030 identifies the particular nodes within thegroup of nodes that are to receive and implement the virtual machineconfiguration. The deployment scheduler 1030 identifies the nodes basedon the statistical data received from the nodes that is then storedwithin the data store 1050. Specifically, the statistical data is usedto identify which nodes have the available resources to host the virtualmachine configuration and which nodes are optimal for hosting thevirtual machine configuration. Accordingly, the deployment scheduler1030 performs a constraint optimization process to identify the one ormore nodes that are best suited to optimally host a virtual machineconfiguration. For example, a first node running with a type 2hypervisor may be less efficient in hosting a user configuration than asecond node running with a type 1 hypervisor based on networkingrequirements of the user configuration and networking performancecharacteristics of the type 1 hypervisor relative to the type 2hypervisor. Therefore, the deployment scheduler 1030 would select thesecond node over the first node even though both nodes may have thesufficient amount of available resources to host the configuration.

The deployment scheduler 1030 then formulates provisioning messages forallocating resources (e.g., disk space, network bandwidth, memory, etc.)to the virtual machine configurations that are to be hosted or arealready hosted by the nodes. These provisioning messages are based on auniform set of messages that the hypervisor management module hasnormalized for all nodes irrespective of the different hypervisors oneach node.

The provisioning messages issued by the hypervisor management modulecause a physical transformation to occur at any node (i.e., computersystem) within the grid of nodes. Specifically, the provisioningmessages generated by the hypervisor management module cause thephysical resources of a node to be allocated to the one or more virtualmachines that are to operate using the set of provisioned physicalresources. More specifically, the provisioning messages allocate a setof physical resources for the exclusive use by a particular virtualmachine. The provisioning of the resources thus restricts other virtualmachines from accessing the provisioned set of resources. In thismanner, each virtual machine is provided with a dedicated set ofhardware resources.

In some embodiments, the provisioning messages cause a physicaltransformation to the processing resources of a node by allocating a setof the processing resources (e.g., processing cycles) to the operationof a particular virtual machine that is hosted by the node. Thisallocation may include increasing or decreasing the processing resourcesthat are already allocated to the particular virtual machine. In someembodiments, the provisioning messages cause a physical transformationto the memory resources of a node by allocating a set of the memoryresources (e.g., block device and random access memory) to the operationof a particular virtual machine that is hosted by the node. Thisallocation may include increasing or decreasing the memory resourcesthat are already allocated to the particular virtual machine. In someembodiments, the provisioning messages cause a physical transformationto the networking resources of a node by allocating a set of thenetworking resources (e.g., networking throughput) to the operation of aparticular virtual machine that is hosted by the node. It should beapparent to one of ordinary skill in the art that the provisioningmessages issued by the hypervisor management module of some embodimentsmay include other provisioning messages for allocating other physicalresources of a node in addition to or instead of those provisioningmessages described throughout this document. Examples of variousprovisioning messages are provided in the subsection below.

To issue the provisioning messages, the deployment scheduler 1030identifies the appropriate hypervisor manager of the hypervisormanagement module 1010, of which only two, 1075 for a Xen® hypervisorand 1080 for a third party hypervisor (e.g., Virtual Iron®), are shownfor the sake of simplicity. However, it should be apparent to one ofordinary skill in the art that several addition hypervisor managers maybe supported by the hypervisor management module 1010 depending on thenumber of hypervisor present in the grid of nodes.

The appropriate hypervisor manager is one that is capable of translatingthe deployment scheduler 1030 defined provisioning messages such thatthey are executable by the corresponding hypervisors on the nodes forwhich the messages are intended. The provisioning messages are then sentto the identified hypervisor manager 1075 or 1080 for translation. Insome embodiments, the hypervisor management module contains a databasestoring the message translations between the normalized set of messagesand those for the various hypervisors supported across the grid ofhosting nodes. Once translated by the hypervisor manager 1075 or 1080 tothe proper format, the hypervisor manager 1075 or 1080 interfaces withthe hypervisor, utility management module, and/or hardware resources ofthe receiving node and submits the message for execution on the node.

The API message processor 1020 of some embodiments also receivesmessages that specify control operations such as starting or stopping anexisting virtual machine as some examples. The hypervisor managementmodule then generates control messages to send to the various nodes.Like the provisioning messages, the control messages issued by thehypervisor management module of some embodiments perform a physicaltransformation on the node receiving the control message. For instance,a control message that specifies that a virtual machine on a particularnode halt its operations will cause the physical resources of theparticular node processing and executing the operations of the virtualmachine to cease operations. Specifically, the processing resources willcease execution of operations for the virtual machine and the memorywill be used to retain the state of the virtual machine until operationsare resumed. Similarly, a control message generated and sent from thehypervisor management module to a particular node specifying that avirtual machine on the particular node commence operations will causethe resources of the node to begin processing and executing theoperations for the virtual machine. As such, the control message issuedby the hypervisor management module of some embodiments facilitatescontrol over resources of the various nodes coupled to the hypervisormanagement module.

The control messages may be specified through, and received from, thefront-end provisioning manager described above with reference to FIG. 9.In such instances, the API message processor 1020 utilizes the virtualmachine controller 1040 to formulate the control messages and toidentify the proper hypervisor manager 1075 or 1080 for translating thecontrol messages such that they are executable by the correspondinghypervisors on the nodes for which the messages are intended. Thecontrol messages are then sent to the identified hypervisor manager fortranslation. Once translated to the proper format, the hypervisormanager interfaces with the hypervisor on the receiving node and submitsthe message for execution by the hypervisor. Examples of various controlmessages are provided below.

As illustrated in FIG. 10, the hypervisor managers 1075 and 1080 includevarious components for performing the translation of provisioning andcontrol messages. As illustrated, the hypervisor managers 1075 and 1080include (1) a virtual machine power switch 1085, (2) a resourceallocator 1090, (3) a configuration manager 1095, and (4) a networkingprovisioner 1097.

In some embodiments, the virtual machine power switch 1085 of thehypervisor manager translates and passes control messages to the nodes.For example, a Xen hypervisor manager contains the logic and protocolsfor modifying control messages generated by the virtual machinecontroller 1040 such that they can be understood and executed by a Xenhypervisor on a specified node receiving the control message.

In some embodiments, the resource allocator 1090 allocates a blockdevice (e.g., disk space) and other processing resources (e.g., memoryand processing units) from one or more nodes to a virtual machineconfiguration. The configuration manager 1095 controls the configurationof the allocated resources. In some embodiments, the configurationmanager 1095 communicably couples with the utility management module(UVM) on the node and specifies the configuration parameter to beconfigured by the utility management module. In some such embodiments,the configuration manager 1095 passes a text file containing theoperating system to be installed, other applications to be installed,and custom parameters for the operating system and applications to bemodified before installation. In some other embodiments, theconfiguration manager 1095 bypasses the utility management module anddirectly configures the allocated resources. This may be the case wherethe user configuration is a static image that requires no change and thehypervisor operating on the node contains the functionality needed toconfigure the resources.

In some embodiments, the network provisioner 1097 is tasked withallocating network resources (e.g., I/O bandwidth, load balancing, etc.)and configuring the network resources (e.g., assigning IP addressed,VLANs, etc.) for one or more nodes to the virtual machine configuration.It should be apparent to one of ordinary skill in the art that thevirtual machine power switch 1085, resource allocator 1090,configuration manager 1095, and networking provisioner 1097 may includesoftware or hardware modules within the hypervisor management module1010. Moreover, additional modules may be used in addition to or insteadof some of these above enumerated modules.

The hypervisor managers 1075 and 1080 also include various interfacesfor directly accessing the utility management modules, the hardwareresources, hypervisors, and 3^(rd) party manager that provide variousinterfaces into some of the nodes. Through these interfaces, thehypervisor management module is able to (1) pass the virtual machineconfigurations, provisioning messages, and control messages directly tothe hardware resources of a node, the hypervisor operating on the node,or to the utility management module on the node and (2) acquirestatistical data about the virtual machines operating on the nodesindirectly from the utility management modules or directly from thevirtual machines. As noted above, the statistical data is retrieved bythe status poller 1060. The statistical data is then normalized by thenormalization engine 1070 before it is stored in the data store 1050.The normalization process is described in further detail below.

The hypervisor management module provides hosting service providers theability to scale hardware resources to virtually unlimited proportions.As noted above, the hypervisor management module seamlessly merges theprocessing resources of physically separate hardware server nodes tocreate a logical amalgamation of resources that can be partitioned inany manner to any number of users irrespective of the differenthypervisors and hardware resources of each node. By also monitoring theusage and availability of resources across the nodes, the hypervisormanagement module is able to optimally provision sets of the resourcesto virtual machine configurations that are modified or created by users.

The hypervisor management module provides such functionality byinterfacing with two or more different virtualization hypervisors, eachhypervisor operating on a different hardware node. In some embodiments,the hypervisor management module includes one or more APIs (e.g., ahypervisor proxy manager) for facilitating the communications with thedifferent hypervisors. The APIs of the hypervisor management module areable to interface and issue messages in order to automaticallyinstantiate, monitor, and remove virtual machines across the differenthypervisors.

As noted above in the Overview section, the different hypervisorsinclude: (1) hypervisors of different types (e.g., type 1, type 2,etc.), (2) hypervisors of different vendor implementations, (3)hypervisors with different messaging interfaces, (4) hypervisors thatsupport virtualization of different sets of guest operating systems, (5)hypervisors that support virtualization of a particular set of hardware,and (6) hypervisors that provide full virtualization, partialvirtualization, paravirtualization, etc.

For different hypervisors of different vendor implementations, thehypervisor management module includes different APIs that interface witheach of the different vendor implementations. For instance, the APIs ofthe hypervisor management module are able to interface with Xen® andMicrosoft's® Hyper-V hypervisors which are both type 1 hypervisors, butare hypervisors implemented differently by different vendors. These APIsconvert the normalized set of messages supported by the hypervisormanagement module of some embodiments to the different messagingprotocols, commands, and interfaces of the different hypervisors.

As mentioned above, the different hypervisors used within the grid ofnodes of some embodiments include hypervisors that supportvirtualization of different sets of guest operating systems. Forinstance, a first hypervisor operating on a first node in the grid ofnodes supports virtualization of Microsoft Windows® and Apple MAC OS®guest operating systems and a second hypervisor operating on a secondnode in the grid of nodes supports virtualization of Linux, UNIX,Solaris, and FreeBSD operating systems. By providing the different APIsto communicate with the different hypervisors, the hypervisor managementmodule further facilitates the ability for the hosting system of someembodiments to support user virtual machines of any commerciallyavailable operating system. This is because each of the differenthypervisors supports virtualization of a specified set of guestoperating systems. Currently, no single hypervisor provides anall-in-one virtualization solution that virtualizes every commerciallyavailable operating system. Therefore, it is necessary for thehypervisor management module of some embodiments to interface withmultiple different hypervisors in order to provide a hosting system thatis able to virtualize and host any such commercially available hostingsystem.

Each of the different hypervisors may also support virtualization ofdifferent sets of hardware. For example, a first hypervisor virtualizesresources of a first node in the grid of nodes that includes an x86based processor and a second hypervisor virtualizes resources of asecond node in the grid of nodes that includes a RISC based processor.Since the first hypervisor is compiled for the x86 based processor, itis incompatible with and cannot be used to virtualize the RISC basedprocessor of the second node. It should be apparent to one of ordinaryskill in the art that the grid of nodes may include hypervisors fornodes with other processors such as an ARM based processor, a MIPS basedprocessor, etc. Furthermore, the nodes may include other hardwareresources that may be virtualized by some but not all of the differenthypervisors.

Additionally, different hypervisors have different performancecharacteristics. This is because different hypervisors provide differentlevels of virtualization. Some hypervisors provide full virtualizationwhereby the guest operating system being hosted does not need to bemodified in any way. Other hypervisors provide partial virtualizationwhereby the virtual machine simulates instances of some of theunderlying hardware platform. For example, this may include simulationof address spaces. Other hypervisors still provide forparavirtualization. In paravirtualization, the guest operating system ofthe virtual machine is modified with a special API that providesspecialized calls to the hardware. It should be apparent to one ofordinary skill in the art that the foregoing provides an exemplary setof different hypervisors and that other different hypervisors exist andare supported by the hypervisor management module of some embodiments.

By supporting many such hypervisors through its various APIs, thehypervisor management module is able to facilitate an automated hostingsystem where the hardware nodes of the hosting system execute withdifferent sets of hardware. Additionally, users of hosting system neednot be restricted to any particular set of operating systems, virtualmachine configuration, etc. as the hypervisor management module of someembodiments hosts any user virtual machine by identifying theappropriate set of hardware, hypervisor, and software needed to host theuser virtual machine.

The hypervisor management module API includes a plug-in interfacewhereby the module is adaptable to support new hypervisors andhypervisor types as they are made available and commercially deployed.In this manner, a new hypervisor may be introduced into the grid ofnodes with the hypervisor management module of some embodiments managingthe resource allocation and virtual machine configurations for thatnode. Additionally, the various modules of the hypervisor managementmodule transform a general purpose computer to a special purposecomputer that automatedly performs the above described functionality ofthe hypervisor management module of some embodiments.

B. Messaging

FIGS. 11 and 12 provide examples of various messages exchanged betweenthe RMS and the hypervisor management module. In some embodiments, themessages are first analyzed by the virtual machine controller or virtualmachine deployment scheduler of the hypervisor management module inorder to determine the node intended as the destination of the message.Additionally, the virtual machine controller or virtual machinedeployment scheduler identifies the hypervisor operating at thedestination node. The message is then forwarded to the appropriate proxymanager of the hypervisor management module where scripts parse theoperation and parameters associated with the operation in order to issuea corresponding command that is executable by the hypervisor on thenode.

FIG. 11 illustrates a control message 1105 for starting the operation ofa virtual machine passed from the RMS 1110 to the hypervisor managementmodule 1120 in accordance with some embodiments. The actual executablecommand is shown at line 1130. In some embodiments, the message 1105 isa series of integer values. The first integer value 1140 identifies thecustomer issuing the command using a customer identification number. Thesecond integer value 1150 is a grid variable for identifying theparticular node on which the virtual machine operates. The third integervalue 1160 is a session identifier for identifying the virtual machine.The fourth integer value 1170 specifies a job number that specifies thecontrol operation to be performed (e.g., start, stop, etc.).

Some embodiments include other control messages for stopping operationof a virtual machine, resuming operation of a virtual machine, rebootinga virtual machine, taking a snapshot of a virtual machine, restoring avirtual machine to a previous snapshot, reporting statistics from avirtual machine, saving an image of a virtual machine, or migrating avirtual machine from one node to another. Stopping operation of avirtual machine may be necessary when up-scaling or down-scalingresources of a virtual machine, changing configuration parameters, orsimply suspending operation during hours of low usage. By haltingoperation of the virtual machine, changes can be made to a virtualmachine and tested before deploying the changes to the public.

Rebooting a virtual machine is often necessary to implement recentchanges that have been made to a virtual machine configuration. In someembodiments, rebooting restarts operation of a virtual machine so thatthe recently implemented changes become incorporated within theoperation of the virtual machine.

Some embodiments utilize the snapshot functionality to instantiatemultiple copies of a single virtual machine that is known to befunctioning at a particular instance in time. Specifically, the snapshotcontrol message creates an image of a user virtual machine at theparticular instance in time. The snapshot includes all functionalcomponents of the virtual machine (e.g., application programs, operatingsystem, hardware requirements) and the configuration of the componentsat the time of the snapshot. From the snapshot, an exact copy of thevirtual machine can then be deployed at another node within the grid ofhardware nodes to meet increased demand. Additionally, the snapshot maybe stored at a database of the hypervisor management module for backuppurposes. Saving an image provides similar functionality where thestored image is the originally deployed image.

FIG. 12 illustrates a provisioning message 1205 for adding a componentto an existing virtual machine operating on a particular node in thegrid of nodes in accordance with some embodiments. In this figure, themultiple parameters are passed in conjunction with the command foradding the component. In this figure, the message 1205 specifies addinga web server to a virtual machine. The message includes a firstparameter 1210 specifying an amount of a block device to allocate fromthe node to the server component, a second parameter 1220 indicating animage with which to configure the component, a third parameter 1230specifying an amount of memory to allocate from the node to the servercomponent, a fourth parameter 1240 specifying an amount of processorresources to allocate from the node to the server component, and variouscustomization parameters 1250 to configure the server component, such asan IP address, hostname, etc. Other parameters are further specified,however for the sake of simplicity are not discussed.

Some embodiments implement additional provisioning messages. Forexample, the hypervisor management module may issue a deleteprovisioning message to remove an already configured resource orcomponent of a user virtual machine. Additionally, a copy provisioningmessage may be issued to duplicate an already configured component,resource, or virtual machine at a single node or across one or moredifferent nodes. Other provisioning messages include messages toupscale/downscale processing resources, upscale/downscale memoryresources, upscale/downscale block device resources, andupscale/downscale network I/O where upscaling involves provisioningadditional resources and downscaling involves removing alreadyprovisioned resources.

In some embodiments, the RMS passes the messages as text files to amessage queue of the hypervisor management module. The hypervisormanagement module runs a cron job or a process that regularly checks themessage queue for newly scheduled messages from the RMS. If a newmessage exists, the hypervisor management module processes the message.Once the messages are parsed by the appropriate proxy managers, thehypervisor management module then conveys the messages to the nodesusing the Secure Shell (SSH) protocol with IP addressing. Accordingly,in some embodiments, the hypervisor management module, each of thenodes, each hypervisor of each node, and each utility management moduleon each node is identified using an IP address.

The hypervisor management module of some embodiments maintains stateinformation for each of the nodes that the hypervisor management moduledirects to perform a given control message or provisioning message. Thehypervisor management module uses the state information to lock accessto certain nodes or virtual machines operating on the nodes. This isbecause some of the messages require exclusive access to the node's orvirtual machine's set of resources. For example, issuing a message totake a snapshot of a particular virtual machine while simultaneouslycreating a new component for the particular virtual machine creates apotentially invalid snapshot image. Similarly, adding multiple serverapplications to a particular virtual machine may exhaust the provisionedset of resources for the virtual machine. However, by maintaining stateinformation for the node, the hypervisor management module may seriallyadd each server application and then monitor the resource usage aftereach such addition in order to prevent the resources from beingexhausted. In some embodiments, the node state information is retainedwithin a database of the hypervisor management module. In otherembodiments, the hypervisor management module sets the state informationfor a particular only after issuing a command to the particular node.The hypervisor management module then removes the state informationafter all provisioning or control operations are complete for theparticular node.

C. Statistics Monitoring

The hypervisor management module of some embodiments retrieves usagestatistics from the hardware nodes. In some embodiments, the hypervisormanagement module retrieves statistics directly from the hardwareresources of the node, from the hypervisor operating on the node, orfrom the virtual machines themselves indirectly through the utilitymanagement module operating on the node. The statistics include dataregarding (1) processor usage (where the processor usage is recorded perprocessing core), (2) memory usage, (3) block input/output usage (e.g.,input/output operations per second (IOPS)), (4) disk usage, (5) type oftraffic passing into and out of a particular node, (6) number of virtualmachines on a node, and (7) configuration of the virtual machines on anode.

In some embodiments, the statistical data gathered by the hypervisormanagement module is further detailed to specify statistics for eachprocess running on a virtual machine. Some examples of these statisticsinclude the memory usage per process, the processor usage per process,the amount of disk storage required per process, and the networkinput/output bandwidth consumed by each process. The memory usagestatistics further include statistics for page table overlap and memoryover commit data.

In some embodiments, the statistical data gathered by the hypervisormanagement module provides more detailed disk usage statistics. Forexample, the detailed disk usage statistics identify the logical volumefor each guest virtual machine, free space per virtual machine, amountof file fragmentation per virtual machine, redundant array ofinexpensive disks (RAID) errors including bad stripes and RAIDperformance metrics such as per disk errors, back blocks, and other suchperformance metrics.

Other statistics monitored by the hypervisor management module includethe total bandwidth throughput per network interface, packets persecond, errors or dropped packets, and statistics per each virtual localarea network (VLAN) tag. Additionally, the hypervisor management modulemay collect statistics regarding the software operating on each of thenodes. For example, the hypervisor management module of some embodimentsgathers statistics regarding operating system performance such as errorsby a UNIX “daemon” and daemon reporting information. Other softwarestatistics include: (1) hypervisor virtual machine events including thenumber of reboot, restart, and stop events, (2) hypervisornetwork-creation events including create, delete, flap virtualinterfaces, and bridging of groups, (3) login and authenticationattempts and successes to an operating system or application programs ofthe virtual machine, and (4) per node control message statistics dataincluding an amount of message per second, failed messages, and numberof retry messages.

By collecting these statistics from multiple nodes, the hypervisormanagement module constructs an overall picture of the resources of thehosting system that includes multiple disparate physical servers. Fromthe collected data, the hypervisor management module is able to reportvirtual machine performance per virtual machine on a node. Additionally,the hypervisor management module uses the statistical information todetermine the availability of resources from which subsequent schedulingand deployment decisions are made for modified or newly created virtualmachines. Moreover, these scheduling and deployment decisions can beoptimized based on the statistics such that utilization of the resourceswithin the grid of nodes is maximized to support as many virtualmachines as possible while guaranteeing a certain level of performancefor each such virtual machine.

The retrieved statistical data is received by a normalization engine ofthe hypervisor management module. The normalization engine is taskedwith (1) receiving statistical data that is formatted differently by thedifferent hypervisors, hardware resources, and virtual machines and (2)formatting the data into a uniform structure. For example, thenormalization engine of some embodiments converts different data unitsof a particular measurement into a single data unit (e.g., bits intobytes). Alternatively, the normalization engine of some embodimentsconverts different data structures into a single data structure, such asconverting HTML formatted data structures into XML. It should beapparent to one of ordinary skill in the art that various othernormalization techniques may be utilized by the normalization engine.

The normalization engine then supplies the normalized data to the datastore for storage and subsequent use by the hypervisor managementmodule. In this manner, the statistical data that is stored within thedata store of the hypervisor management module is uniformly formattedfor all nodes irrespective of the different types of hypervisors on thenodes or the different hardware and software resources of the nodes.

Additionally, by being able to directly acquire the statistical datafrom the utility management modules, virtual machines, and the hardwareresources of the nodes, the hypervisor management module is able tocircumvent the different virtualization databases of the hypervisorsthat may be implemented differently with different schemas, technology,and statistical parameters. In this manner, some data may be acquiredfrom the various hypervisors, but should a first hypervisor includecertain statistical parameters that a second hypervisor does notinclude, the hypervisor management module may directly interface withthe resources of the nodes to acquire the missing parameters.Accordingly, the data store provides a single schema instance fromstatistical data that may arrive from multiple different schemainstances.

FIG. 13 presents a process 1300 for collecting statistical data from thegrid of nodes in accordance with some embodiments of the hypervisormanagement module. The process 1300 begins by selecting (at 1310) a nodein the grid. The process then interfaces (at 1320) with the node inorder to retrieve statistics using either the hypervisor, the directinterface to the resources of the node, the interface to the utilitymanagement module, or by using two or more such interfaces. Theretrieved data is then normalized (at 1330) before it is stored (at1340) where the data is combined with the statistical data of othernodes in order to generate normalized statistical data for all nodes inthe grid. The process determines (at 1350) whether additional nodes needto be monitored. If so, the process selects the next node and retrievesthe data as described above. Otherwise, the process generates (at 1360)a mapping for the resource usage and available resources of the grid ofnodes.

The mapping may be used internally to facilitate the automatedscheduling and deployment of subsequently modified or added virtualmachines. Moreover, the statistical data may be presented to one or moreusers through a graphical interface to display a state of a user'svirtual machine. This provides the user with an updated view of theoperating condition of his or her virtual machine. From the view, theuser is able to modify the virtual machine's configuration if needed(e.g., add servers, remove servers, allocate additional physicalresources, etc.).

In some embodiments, the collected data assists in identifyingbottlenecks within the user configuration. For instance, a first webserver of a particular configuration allocated to a first node in thegrid with only a minimal allocation of memory may be causing a secondweb server of the particular configuration allocated to a second node inthe grid to lose performance. By retrieving and analyzing theseperformance statistics, some embodiments notify the user of thebottlenecks and allow the user to allocate additional resources toameliorate the bottleneck.

In some embodiments, the process 1300 is continually run to provide areal-time view of the resource usage and availability. In some otherembodiments, the process 1300 is run at specified intervals or is runover only a set of nodes. For instance, a particular user's virtualmachine configuration may be deployed over three disparate nodes. Inorder to determine whether additional components may be allocated to theuser's configuration, the hypervisor management module first monitorsthe three nodes. If sufficient available resources exist on the nodes,then the additional components are deployed to the nodes. Otherwise, theadditional components must be deployed to other nodes.

D. Resource Allocation

In addition to monitoring resources of the nodes, the hypervisormanagement module of some embodiments controls the allocation andoperation of the various virtual machines on the nodes. In some suchembodiments, the hypervisor management module starts, stops, adds, anddeletes virtual machines from the nodes. For instance, some embodimentsof the hypervisor management module specify commands for adding avirtual machine configuration to one or more nodes. To do so, thehypervisor management module first allocates the necessary resourcesfrom the nodes using one or more of the above described provisioningmessages.

FIG. 14 presents a process 1400 performed by a hypervisor managementmodule of some embodiments for the allocation of resources for a newlyspecified virtual machine configuration. The process 1400 begins byreceiving (at 1410) a virtual machine configuration that is passed fromthe front-end logic to the hypervisor management module. The processthen identifies (at 1420) at least one node within the grid of nodes tohost the virtual machine configuration. In some embodiments, the one ormore nodes can be identified by the resource management system of someembodiments such that the hypervisor management module simply performs adeployment of the virtual machine according to requirements specifiedwithin the virtual machine configuration. In other embodiments, thehypervisor management module identifies the one or more nodes based onresources requirements specified within the virtual machineconfiguration.

The process interfaces (at 1430) with the hypervisor of each identifiednode by identifying the proper set of API calls for communicating withthe node. Since the hypervisors directly control the allocation ofhardware resources for their respective nodes, the process issues (at1440) a command to the one or more hypervisors to allocate a set of theidentified nodes' resources to the virtual machine configuration.

Each hypervisor performs a mapping from the hypervisor's correspondingnode physical resource space to a logical resource space allocated forthe virtual machine configuration. The virtual machine receives the oneor more logical mappings as a single resource space over which tooperate. The virtual machine is provided exclusive use of the mappedresources such that operation of other virtual machines on the same nodedoes not affect the performance of the newly created virtual machine. Inthis manner, some embodiments guarantee that an amount of processingresources is always available to the virtual machine.

In some embodiments, allocating the set of resources includes allocatingone or more of the following processing resources to the virtual machineconfiguration: processor cycles, memory, block device (e.g., diskstorage), network interfaces, network bandwidth, backup devices, andstandalone devices such as firewalls and load balancers. It should beapparent to one of ordinary skill in the art that some embodimentsallocate additional resources in addition to the resources enumeratedabove. Each of the various resources can be allocated in differentamounts according to virtual machine configuration. Additionally, thenodes from which the set of resources are allocated need not reside at asingle destination, but may instead be distributed across multipledifferent server farms at multiple destinations. Accordingly, thephysically separate nodes in the grid of nodes logically appear as aunified grid of nodes.

With the resources allocated, the process then facilitates theconfiguration of the resources. The process does so by passing (at 1450)the specified virtual machine configuration and any additionalconfiguration parameters to the corresponding utility management modulesoperating within each of the nodes. As will be described in Section IVbelow, the utility management modules perform the custom softwareconfiguration of the virtual machine configuration.

The process stores (at 1460) the allocation of resources for the virtualmachine configuration into a database tracking the allocation ofresources for the entire grid. In some embodiments, the process alsoreceives the customized configuration data from the utility managementmodules to store within the database. The stored data permits thehypervisor management module to quickly access information for anyvirtual machines distributed across the entire grid of nodes and notjust on a single node. Such information can be used by the hypervisormanagement module of some embodiments to identify and then to remove,migrate, modify, stop, or resume one or more components from anyconfiguration operating within the grid of nodes.

It should be apparent to one of ordinary skill in the art that theprocess 1400 of FIG. 14 is similarly applicable to the removal ordeallocation of resources for a virtual machine configuration. Forexample, the hypervisor management module receives a virtual machineconfiguration or component to remove. The hypervisor management moduleidentifies the one or more nodes and the set of resources allocated forthe operation of the virtual machine operation. The hypervisormanagement module interfaces with the hypervisors of the identifiednodes to cause the hypervisors to remove the virtual machineconfiguration and free the resources allocated to the configuration.Additionally, the hypervisor management module in some embodimentscauses the corresponding utility management modules of the identifiednodes to destroy the data from the file systems as is described infurther detail below.

E. Optimized Resource Allocation

Some embodiments modify the process 1400 of FIG. 14 such that thehypervisor management module optimizes the allocation of resources tomaximize resource usage and to maximize performance of the virtualmachines. FIG. 15 presents a process 1500 implemented by the hypervisormanagement module of some embodiments to optimally allocate resourcesfrom the grid of nodes to a virtual machine configuration.

As in process 1400, the process 1500 begins by receiving (at 1510) avirtual machine configuration that is passed from the front-end logic tothe hypervisor management module. The process then identifies (at 1520)all nodes within the grid of nodes with sufficient resources to host thevirtual machine configuration.

The process analyzes (at 1530) the identified nodes based on a set ofconstraint optimization criteria to determine (at 1540) the optimal nodefrom the identified nodes for hosting the configuration. In someembodiments, the constraint optimization criteria consider variousfactors of the nodes in order to decide which of the identified nodes isan optimal node.

For example, different nodes may have different hypervisors with eachhypervisor executing certain operations more efficiently than otherhypervisors. A first operating system specified in a first configurationmay perform more efficiently using a node running Xen® virtualization,whereas a second operating system specified in a second configurationmay perform more efficiently using a node running Parallels®virtualization. More specifically, a type 1 hypervisor (e.g., Xen®) withdirect access to the hardware resources processes I/O requests moreefficiently than a type 2 hypervisor since the type 1 hypervisor doesnot have to traverse through the OS stack before accessing the hardwareresources. Accordingly, a virtual machine configuration that is heavilyI/O dependent would be best served by a node operating with a type 1hypervisor.

As another example, two nodes may be identified as having sufficientresources to host a particular configuration. A first node has 512megabytes (Mb) of random access memory (RAM) available for allocationand a second node has 100 Mb of RAM available for allocation. If theparticular configuration requires 90 Mb, the process may identify thesecond node as the most optimal node so that the resources of the secondnode are maximized with a larger amount of resources of the first nodebeing available for hosting a configuration requiring a largerallocation of memory.

It should be apparent to one of ordinary skill in the art that thehypervisor management module of some embodiments performs other types ofintelligent resource allocation for a modified virtual machineconfiguration. For instance, when a user specifies adding a particularsoftware component, some embodiments identify nodes in the grid alreadyconfigured with the proper licenses to run the software. In this manner,the hypervisor management module is able to reduce costs for hostedvirtual machines.

Once the optimal node is identified, the process allocates (at 1550) thenecessary resources for the configuration and updates (at 1560) theresource usage mapping for the nodes to indicate that the allocatedresources are no longer available for use in hosting subsequentconfigurations.

The hypervisor management module of some embodiment also efficientlyallocates resources when a virtual machine configuration is modified.FIG. 16 presents a process 1600 performed by the hypervisor managementmodule for modifying an existing virtual machine configuration for avirtual machine that operates within the grid of nodes. The process 1600begins by receiving (at 1610) a modification to an existingconfiguration of a particular virtual machine. In some embodiments,users are able to modify their configurations at any time through thefront-end user interface. In some embodiments, user configurations areautomatically modified by the hosting system to meet increased ordecreased demand in resource usage.

The process identifies (at 1620) one or more hypervisors for the nodesover which the particular virtual machine operates. The process thendetermines (at 1630) whether the modification can be made to theidentified nodes. In some embodiments, a modification involvesallocating additional hardware resources such as network bandwidth ormemory to a configuration. Additionally, the modification may involveadding additional virtual machine components such as a web server,database, cloud storage, or load balancer. If the identified nodes havesufficient unused resources, then the modification can be made on thesenodes. In such instances, the hypervisor management module modifies (at1640) the particular virtual machine by allocating additional resourcesof the existing nodes to the particular virtual machine.

In some instances, the process determines (at 1630) that the existingnodes over which the particular virtual machine operates do not haveadditional resources that may be allocated to the virtual machine.Accordingly, the hypervisor management module must migrate theconfiguration to a different set of nodes with sufficient resources forhosting the modified configuration.

In some embodiments, the hypervisor management module performs anintelligent allocation of resources from the other nodes using theconstraint optimization analysis of process 1500. This is done in orderto provide consistent performance when adding resources to a virtualmachine. Accordingly, the process identifies (at 1650) one or more nodeswith the same hypervisors as the existing virtual machine configurationand sufficient free resources for hosting the modification to thevirtual machine. Once identified, the process allocates (at 1660) thenecessary resources for the modified configuration from the newlyidentified set of nodes. The process then migrates (at 1670) themodified configuration onto the newly allocated nodes where operation ofthe modified virtual machine resumes.

IV. Utility Management Module

In some embodiments, the utility management module, also referred to asthe utility virtual machine (UVM), compliments the functionality of thehypervisor management module by configuring resources that have beenallocated by the hypervisor management module for a virtual machine.Specifically, the utility management module customizes a softwareconfiguration per the specifications of a virtual machine configurationbefore applying the configuration to the allocated resources. Each nodewithin the grid of nodes includes a utility management module forconfiguring all virtual machines operating within the same node as theutility management module.

FIG. 17 presents a grid of sixteen nodes with three nodes 1710, 1720,and 1730 expanded to illustrate the utility management module andvarious virtual machines running on each of these nodes. Each utilitymanagement module is responsible for retrieving and passing thenecessary configuration data to the virtual machines operating on thesame node as the utility management module. By including a utilitymanagement module on each node, the hosting system avoids a bottleneckthat would otherwise exist if a single entity within the hosting systemwas responsible for the configuration of all nodes in the grid of nodes.Instead, each node is independently configured with respect to othernodes by its own utility management module.

Also, by distributing utility management modules across each of thenodes, some embodiments remove a single point of failure for the nodes.If any single utility management module or node fails, all other utilitymanagement modules and nodes continue operating unaffected by the loss.Such an architecture allows for resources to be automatically divertedfrom a failed node to other operational nodes, minimizing downtime forvirtual machines on the failing node. This is made possible by storingthe virtual machine configurations in a database (e.g., data stores 950or 955 of FIG. 9) that is not local to any particular node such that theconfiguration data for the virtual machines on a failing node can beretrieved. Moreover, since each node has its own local storage, someembodiments avoid the single point of failure potentially experienced byhosting services that utilize a storage area network (SAN) in which acentralized storage solution provides storage for all nodes and virtualmachines.

In some embodiments, the utility management module is itself a virtualmachine with an operating system that executes software processesrunning within the utility management module's own virtualized set ofresources of a node. In some embodiments, the software processesexecuted by the utility management module are defined by a set ofscripts located on a remote file system. The hypervisor managementmodule directs each utility management module to execute one or more ofthe scripts at specified instances in time through the various controlprovisioning messages described above. For example, the hypervisormanagement module causes the hypervisor on a node to allocate resourcesfor a virtual machine and then the hypervisor management moduleidentifies one or more scripts for the utility management module toexecute. The scripts cause the utility management module to configurethe allocated resources per a user's configuration using the variousprovisioning messages and control messages described above.

To access the set of scripts, the utility management module mounts theremote file system. It should be apparent to one of ordinary skill inthe art that in some embodiments the scripts are locally stored in thefile system of the utility management module.

A hypervisor of a node therefore allocates a portion of the node'savailable resources for the operation of the utility management module.In this manner, the utility management module of some embodimentstransforms a node from a general purpose computer to a special purposecomputer once the utility management module begins operations on thenode. The special purpose computer is then used by a hosting systemoperator to facilitate the automatic hosting of virtual machines acrossa grid of nodes.

The utility management module performs fully automated processes thatconfigure the file system for each virtual machine based on a virtualmachine configuration specified by a user using the front-end interfacesof some embodiments. Rather than uniformly configuring virtual machineswith static configuration images, the customized configuration of someembodiments removes unnecessary or unused data and only configures thevirtual machines with data related to the user-specified configuration.Moreover, parameters within the software configuration may be customizedby the utility management module in the manners described below withreference to FIG. 22.

A. Utility Management Module Architecture

FIG. 18 illustrates various functional components of a utilitymanagement module 1805 for configuring resources allocated for a virtualmachine configuration in accordance with some embodiments of theinvention. In this figure, the utility management module 1805 includes:(1) a configuration manager 1810, (2) a partitioning module 1820, (2) aformatting module 1830, (3) a software configuration module 1840, (4) anetwork configuration module 1850, (5) a status poller 1860, (6) aresource modifier 1870, (7) a replicator 1873, and (8) a configurationstore 1876. As noted above, the utility management module 1805 is avirtual machine that operates concurrently with the other virtualmachines on a node. The various managers and modules (e.g., 1810-1873)of the utility management module 1805 execute one or more scripts asdirected by the hypervisor management module to transform a set of thegeneric computing resources of a node into a specific machineimplementation (i.e., the utility management module 1805) forautomatedly configuring and instantiating other virtual machines on thesame node.

The modules of the utility management module 1805 will be described inconjunction with the process 1900 of FIG. 19. FIG. 19 presents a process1900 performed by a utility management module to configure an allocatedset of resources based on one or more scripts in accordance with someembodiments. The process 1900 begins by the configuration manager 1810of the utility management module 1805 receiving (at 1910) auser-specified virtual machine configuration and associatedconfiguration parameters for configuring a set of hardware resourcesallocated for hosting a virtual machine. In some embodiments, theuser-specified virtual machine configuration is passed from thehypervisor management module to the configuration store 1876. In someembodiments, the configuration store 1876 specifies a directory within afile system of the utility management module 1805. The configurationmanager 1810 then runs a script or process that periodically checks theconfiguration store 1876 for new configurations or modifiedconfigurations.

When a new or modified configuration is found, the configuration manager1810 causes operations 1920-1990 to occur. In this manner, theconfiguration manager 1810 controls the operation of the functionalcomponents of the utility management module 1805. Specifically, theconfiguration manager 1810 parses through the user-specifiedconfiguration and identifies the various scripts to execute. One suchscript invokes the partitioning module 1820 of the utility managementmodule 1805.

The partitioning module 1820 partitions (at 1920) disk storage resourcesof a block device that have been allocated by the hypervisor managementmodule 1875 for the virtual machine configuration. The partitioningcreates the one or more file systems to store the configuration filesand user data for the user specified virtual machines. For instance, avirtual machine configuration that specifies two independently operatingweb servers will partition the allocated disk storage into twopartitions 1880, one for each web server. Additionally, the utilitymanagement module 1805 contains logic to optimally provide thepartitions such that a first web server that is expected to experiencesignificant load and service a majority of user requests through severalapplication programs is partitioned with more resources (e.g., a largerdisk partition) than a second web server that is expected to experienceonly minimal load. The partitioning can also be based on an amount ofresources that are requested by a user such that users customize thesize of each of their respective partitions through the front end logic.In some embodiments, the partitioning involves creating a bootpartition, data partition, and/or swap space for a virtual machine. Insome embodiments, the partitioning module 1820 similarly partitions alocal temporary file system of the utility management module that isused to construct the virtual machine before the virtual machine iscopied to the actual file system allocated to the virtual machine.

It should be apparent to one of ordinary skill in the art that eventhough the disk partitions 1880 for virtual machine 1885 are shownseparate from the hardware resources 1865 of the node, the partitionsrepresent a portion of the hardware resources that have been allocatedto the virtual machine 1885. Other shared resources (e.g., CPU,networking I/O, etc.) or unused resources are represented by the pool ofhardware resources 1865.

After partitioning the block device, the configuration manager 1810facilitates the formatting of the resources using the formatting module1830 of the utility management module 1805 and a script retrieved fromthe remote file system that defines the formatting operations performedby the formatting module 1830. The formatting module 1830 formats (at1930) the partitioned resources. Formatting defines the organization orstructure of the virtual machine file system. For example, depending onthe type of formatting, different file system may be created with blocksof different sizes (e.g., 512 bytes, 1 kilobyte, 4 kilobytes, etc.). Thedifferent sized blocks represent the smallest amount of the actual blockdevice that can be allocated to hold a file or directory. By providingcontrol over the formatting of the file system, some embodiments of theutility management module create customized file systems that aredefined differently based on the user specified virtual machineconfiguration. Such customization is ideal in controlling the amount offragmentation, speed, and storage overhead that result from theformatting characteristics of the file system.

In some embodiments, the formatting module 1830 formats the actual filesystem of the virtual machine and also the local temporary file systemof the utility management module that is used to construct the virtualmachine. Different configurations may require different formatting ofthe partitioned resources. For example, a first component in the virtualmachine configuration may require FAT32 formatting for its partitionedset of resources and a second component in the virtual machineconfiguration may require another file system formatting (e.g., NTFS,FAT16, etc.) for its partitioned set of resources. In some embodiments,the image to be installed is a self-booting image that need only becopied onto the allocated file system. In some such embodiments, theconfiguration manager 1810 bypasses the formatting of the resources.

In some embodiments, the formatting module 1830 installs specific blockdevice drivers that provide access to the disk storage device of thenode. Specifically, the formatting module 1830 installs the block devicedrivers to facilitate disk access for the user specified operatingsystem image that is to be used for the virtual machine. Accordingly,the formatting module 1830 analyzes the user virtual machineconfiguration to determine the appropriate set of block device driversto install to the file systems before the software configuration module1840 installs the OS image.

Once the partitioning and formatting of the resources are complete, theconfiguration manager 1805 tasks the software configuration module 1840with constructing the virtual machine image based on one or more scriptsfor creating the user's custom configuration. The software configurationmodule 1840 configures the local temporary file system of the utilitymanagement module according to the user specified parameters. Thesoftware configuration module 1840 retrieves, customizes, and installsonto the temporary file system the software configuration specifiedwithin the virtual machine configuration. The software configurationmodule 1840 retrieves (at 1940) an operating system software image(e.g., Windows Vista®, Linux, or UNIX) from an OS image store 1890 basedon the virtual machine configuration. The software configuration module1840 also retrieves (at 1940) the various application programs specifiedwithin the virtual machine configuration from an applications imagestore 1895. These application programs may include any commerciallyavailable software application as well as user provided proprietarysoftware applications. It should be apparent to one of ordinary skill inthe art that the retrieved OS image may include various applicationprograms. As such, different images may exist within the image store forthe same OS, where each image includes different application programs(e.g., Windows Server 2003 with an SQL database and Windows Server 2003with an APACHE Web Server).

In some embodiments, the software images include default configurationparameters for each software component. These images are static imagesintended for generalized application and for providing a template fromwhich the utility management module is able to customize the virtualmachine based on the user-specified configuration for the virtualmachine. Therefore, the utility management module identifies a bestmatching image for a virtual machine based on the user's virtual machineconfiguration.

The software configuration module 1840 decompresses (at 1950) the imagelocally at the temporary file system if the image is compressed. Byretrieving a compressed image from the data store, some embodimentsreduce the data that is transferred across the hosting system in orderto minimize bottlenecks at the data store or elsewhere in the system.

The software configuration module 1840 then installs (at 1960) theoperating system with any specified customizations, deploys the kernel,and installs and customizes any other software that is specified for thevirtual machine configuration per the virtual machine configuration tothe temporary file system of the utility management module. In someembodiments, installing the operating system involves also making theoperation system bootable. Modifying and/or customizing an image isdescribed in further detail with reference to FIG. 22 below.

As noted above, since each node directly retrieves images from the imagestores, some embodiments avoid having a single point of failure impactthe performance of the overall hosting system. A further benefit ofhaving the utility management modules retrieve and install the images isthat the instantiation of a virtual machine occurs locally on a node.The result is a distributed system and method of virtual machineconfiguration as each utility management module on a particular node cansimultaneously or in parallel instantiate a virtual machineconfiguration with other utility management modules of other nodes. Eachnode is therefore not dependent on other nodes. As such, the hypervisormanagement module can assign multiple instantiations of virtual machinesacross the utility management modules of multiple nodes at any timewithout having to wait until each node finishes installing an assignedconfiguration.

Referring back to FIG. 19, in some embodiments, steps 1940-1960 areperformed from a batch process that is executed by the allocated set ofresources. In such instances, the batch process is a script that definesthe various components to be installed which are then retrieved directlyfrom the image store by the batch process.

Once the images are installed, the configuration manager 1810 theninvokes the network configuration module 1850 of the utility managementmodule 1805 to configure (at 1970) the networking resources.Specifically, the network configuration module 1850 installs andconfigures the networking drivers for the virtual machine on aparticular node. For instance, the network configuration moduleconfigures IP addresses, netmasks, ports, load balancing, etc. for thevirtual machine. In some embodiments, the network configuration module1850 performs other network related tasks such as acquiring a DHCPlease.

At this stage the utility management module will have constructed thevirtual machine according the user-specified requirements of the virtualmachine configuration. However, the constructed virtual machine resideswithin the temporary file system of the utility management module.Therefore, the process then uses the replicator 1873 to copy virtualmachine from the temporary file system of the utility management moduleto the file system that has been allocated for the virtual machine. Theconfiguration manager 1810 then instructs (at 1980) the virtual machineto begin execution and the process 1900 ends.

The utility management module 1805 also includes a status poller 1860.In some embodiments, the status poller 1860 directly monitors theperformance and allocation of the hardware resources on the particularnode in which the utility management module 1805 operates. In someembodiments, the data gathered by the status poller 1860 is passed tothe status poller of the hypervisor management module. The status pollerof the hypervisor management module may then aggregate the statisticaldata in order to form an overall picture of the resource usage andperformance of the grid of nodes as opposed to each individual node.

The resource modifier 1870 of the utility management module 1805compliments the resource allocation functionality of the hypervisormanagement module. In some embodiments, the provisioning messages fromthe hypervisor management module are received at the resource modifier1870. The resource modifier 1870 may then process the provisioningmessages and allocate the desired set of resources (e.g., processingcycles, memory, disk storage, etc.) in conjunction with the hypervisoroperating on the same node as the utility management module.

As noted above, the utility management module performs the variousconfiguration and modification operations to new or previously installedconfigurations using a set of automated scripts. According to the set ofscripts, the utility management modules execute system level commandsthat result in the transformation of the computer readable medium of thenode hosting the virtual machines. As noted above, some examples of thephysical transformation include (1) the partitioning and formatting ofhardware resources, (2) the retrieving, modifying, and installing of thecustom software configurations onto the allocated resources of a node toinstantiate the execution and processing of the virtual machine usingallocated resources of the node, (3) the establishing of networkinginterfaces to facilitate communications for the virtual machines, and(4) the deleting, migrating, and encrypting of the virtual machinesoperating on the nodes.

In some embodiments, the partitioning and formatting of hardwareresources performs a physical transformation on the computer readablemedium by creating a file system on the block device that is used tostore files according to a particular format and by designating variouspartitions for the file system. In some embodiments, the retrieving,modifying, and installing of the custom software configuration performsa physical transformation on the computer readable medium by modifyingthe contents of the computer readable medium. In some embodiments, theestablishing of network interfaces performs a physical transformation onthe node by configuring the node in a manner that enables the node tocommunicate with other nodes. In some embodiments, deleting, migrating,and encrypting virtual machines also cause physical transformation tooccur on the node. For example, deleting a virtual machine causes thecontents of the virtual machine to become unreadable on the computerreadable medium unless a readily available backup copy of the virtualmachine was made. Similarly, migrating a virtual machine causes thevirtual machine to execute operations on a different node using adifferent set of hardware resource. Additionally, migrating the virtualmachines causes previously used resources of a first node to be freed(e.g., processing resources, memory, disk, etc.) and previously unusedresources of a second to node be allocated and used in the execution andprocessing of a virtual machine that was migrated from the first node tothe second node. Encrypting a virtual machine causes a file system ofthe virtual machine to become unreadable unless the proper decryptionkeys are known.

FIGS. 20A-20E provide a more detailed description of a utilitymanagement module of some embodiments, as well as some operationsperformed by this module to configure a virtual machine according to auser-specified configuration. FIG. 20A illustrates a node 2005 from agird of nodes of a hosting system of some embodiments. The node includesa utility management module 2045. The utility management module 2045includes: (1) modules and managers 1810-1873, which were described aboveby reference to FIG. 18, (2) the configuration store 2020, and (3) atemporary virtual machine file system 2025. The node 2005 also has anamount of storage 2030 that has been allocated to a virtual machine 2035and other available storage 2070, which represents unused portions ofthe block device of the node that may be allocated to other virtualmachines. In some embodiments, these storage devices 2020, 2025, 2030and 2070 each correspond to one or more physical storage devices. Insome other embodiments, each of these storage devices 2020, 2025, 2030and 2070 is a partition of one or more physical storage devices of thenode 2005. It should be apparent to one of ordinary skill in the artthat the dashed line connecting the other available storage 2070represents that an amount of storage resources are allocated to thevirtual machine. It should also be apparent to one of ordinary skill inthe art that the various managers and modules 1810-1873 of the utilitymanagement module 2045 are stored within a file system storage of theutility management module 2045. Each of the modules or managers1810-1873 includes executable instructions that the utility managementmodule 2045 executes using processing resources of the node 2005.

The configuration store 2020 stores a user-specified virtual machineconfiguration. In some embodiments, the hypervisor management module2065 passes the virtual machine configuration to the utility managementmodule 2045 after the hypervisor management module 2045 has allocatedthe necessary resources from the node 2005. As noted above, theconfiguration manager of the utility management module 2045 periodicallychecks the configuration store 2020 to identify a new or modifiedconfiguration that is to be configured for a particular virtual machineon the node 2045.

The temporary virtual machine file system 2025 is used by the utilitymanagement module 2005 to construct the file system of a virtual machineaccording to the user-specified configuration. In some embodiments, thetemporary virtual machine file system 2025 is separate from the actualblock device 2030 allocated to the virtual machine 2035 by thehypervisor management module 2065. The temporary virtual machine filesystem 2025 thus stores a working copy of the virtual machine that whencomplete and functional is copied over to the block device 2030allocated to the virtual machine 2035.

In some embodiments, the separation is necessary as the virtual machineis provisioned a large amount of storage where only a small fraction ofthis provisioned storage is used to store the virtual machineconfiguration. The remaining storage may be used for user data.Accordingly, by separating the temporary virtual machine file system2025 from the actual allocated block device 2030 for the virtual machine2035, the size of the temporary virtual machine file system 2025 mayinclude a much smaller set of storage resources than the allocated blockdevice for the actual virtual machine.

The other storage 2070 stores other data stored on the node that havenot been provisioned for use by the virtual machine to be configured. Insome embodiments, the other storage 2070 stores virtual machineconfigurations not mentioned with reference to FIGS. 20A-20E, as well asunallocated storage of the node 2005.

FIG. 20B illustrates the node 2005 of FIG. 20A (1) after thepartitioning module of the utility management module 2045 partitions thetemporary virtual machine file system 2025 and the actual block deviceof the virtual machine 2035 as per step 1920 described above and (2)after the formatting module of the utility management module 2045formats the temporary virtual machine file system 2025 and the actualblock device 2030 of the virtual machine 2035 as per step 1930 describedabove. At this stage, the utility management module 2045 has performed afirst set of customizations to the file system 2030 of the virtualmachine 2035. Namely, creating user-specified partitions anduser-specified formatting of the partitions.

FIG. 20C illustrates the node 2005 after the software configurationmodule retrieves and installs an operating system image and zero or moreapplication images onto the temporary virtual machine file system 2025as per steps 1940 and 1950 above. The software configuration module thencustomizes the images according to the user-specified configuration asper step 1960 above. In some embodiments, the customizations includemodifying the images to configure networking parameters for the virtualmachine. FIG. 20D illustrates the node 2005 after the softwareconfiguration module customizes the retrieved set of images per step1960 above.

The temporary virtual machine file system 2025 now stores a complete andfunctional user-specified virtual machine. As shown in FIG. 20E, thereplicator 1873 of the utility management module 2045 copies thecontents of the temporary virtual machine file system 2025 to the actualblock device 2030 allocated to the virtual machine 2035. The virtualmachine can then be instantiated by calls to the hypervisor. In somesuch embodiments, the replicator 1873 then securely deletes the contentsof the temporary virtual machine file system 2025.

Additionally, the utility management module 2045 of some embodimentsnotifies the hypervisor management module 2065 that the configurationhas been installed on the node 2005 (i.e., installed on the hypervisorof the node 2005 in some embodiments). In some embodiments, thenotification includes a copy of the customized virtual machine imagesuch that the image is stored for later access or modification. In someembodiments, the hypervisor management module 2065 informs a front-endprovisioning manager (e.g., the front-end provisioning manager 920 ofFIG. 9) that the virtual machine configuration has been installed on thenode 2005. The front-end provisioning manager then presents anotification to a user (e.g., through a service request server, such asthe service request server 910 of FIG. 9). In some embodiments, thisnotification is a visual notification presented in a GUI (e.g., a GUI ofa standalone application or a web browser).

To partition the file systems of the virtual machines, format the filesystems of the virtual machines, and duplicate the customized images tothe allocated block device of the virtual machines, the utilitymanagement module and its various modules and managers are alloweddirect access to the file system resources of every virtual machineallocated to the same node as the utility management module. In someembodiments, directly accessing the file system includes system callsmade to the hypervisor that then authorize the access. FIG. 21conceptually illustrates the ability of a utility management module ofsome embodiments to access file systems of other virtual machinesoperating on the same node as the utility management module.

In FIG. 21, a node 2110 includes a hypervisor 2115, a utility managementmodule 2120, and several user virtual machines 2130-2140 that performoperations using the hardware resources of the node 2110. As shown, theutility management module 2120 is able to access the file systems ofeach of the virtual machines 2130-2140. However, the virtual machines2130-2140 are prevented from accessing the file systems of the utilitymanagement module 2120 or the file systems of other virtual machines.

Specifically, the hypervisor 2115 maps a set of resources that includesome amount of block storage from the block device of the node 2110 toeach virtual machine 2130-2140. The allocated storage can then beconfigured in any manner to store the virtual machine configurationfiles, such as operating system files, application programs, etc., andother data used in the operations of the virtual machines, such ascustomer specific data. However, since each virtual machine 2130-2140 isonly aware of those resources to which it is allocated, the virtualmachines 2130-2140 are unable to accidentally overwrite data of anothervirtual machine. Any call to access a file system or a portion of a filesystem outside of the set of resources mapped to a particular virtualmachine is intercepted by the hypervisor 2115 and prevented from beingexecuted.

As noted above, the utility management module 2120 is also a virtualmachine, but the utility management module 2120 requires access to theother file systems. Therefore, in some embodiments, node 2110 and othernodes used within the grid of nodes of the hosting system provider ofsome embodiments are modified with a specialized set of instructionsthat permit the utility management module 2120 to access each of thefile systems of each virtual machine 2130-2140 and thus perform variousfunctions over the file systems.

In some embodiments, the hypervisor 2115 operating on the node 2110 isalso modified to recognize the calls and provide the utility managementmodule 2120 special access to file systems of the different virtualmachines. For example, the utility management module 2120 is providedwith administrator level access permissions and each of the virtualmachines 2130-2140 are provided only user level access permissions.

Using the file system access, the utility management module configuresthe allocated resources for a given configuration by adding, removing,or modifying files to and from the file system in order to instantiatethe custom configuration. For example, the utility management module mayuse a resize command for an Ext3, ntfs, or lvm disk partition to modifya configuration by adding additional disk space to the configuration. Insome embodiments, access to the file system is made by the utilitymanagement module in the same manner that the hypervisor of the nodeaccesses the file system.

Several advantages result from having the utility management moduleoperate at the same levels as the other virtual machines of a node. Forexample, by having one virtual machine such as the utility managementmodule configure the file systems of other virtual machines, the hostoperating system of a node (e.g., Domain 0 of a type 2 hypervisor) isfreed from having to devote processing, memory, and storage resourcesthat would otherwise be required to configure the file systems of thevirtual machines. Additionally, the host operating system would become abottleneck. Specifically, some virtual machines would compete forconfiguration or modification resources of the host operating systemwhile other virtual machines would compete for traditional hypervisingoperations that require input/output processing. A further bottleneck isavoided as the host operating system is at a different level ofabstraction than the virtual machines. As such, the host operatingsystem would be unable to directly issue commands and perform operationson the file systems of the virtual machines, whereas the utilitymanagement module is able to do so. Therefore, in some embodiments, theutility management module includes its own operating system andapplication programs as shown above. In this manner, the utilitymanagement module is able to access and modify file systems that wouldotherwise be unable to standalone virtualized applications that are notstandalone virtual machines.

B. Customizing Configurations

FIG. 22 illustrates the various customizations performed by a utilitymanagement module in accordance with some embodiments of the invention.In this figure, a particular node is shown at three different stages ofoperation 2210, 2220, and 2230 with each stage configuring an additionalvirtual machine to operate using the allocated sets of resources. Insome embodiments, each stage is dependent on the other stages andtherefore the stages are performed sequentially. In other embodiments,the stages are independent of one another and therefore may occur inparallel.

At the first stage of operation 2210, the utility management module 2240configures a first allocation of resources 2250 represented by thedashed lines. To perform the configuration, the utility managementmodule 2240 retrieves a first OS image 2270 from an OS image store 2245.The utility management module 2240 then customizes the retrieved imagebased on the user's specification within the virtual machineconfiguration. A first level of customization reduces the number ofcomponents from the retrieved image to produce the customized image2275. For example, the user's virtual machine configuration specifieswhich components are extraneous. If the user's virtual machineconfiguration only specifies a virtual storage server and the retrievedimage also includes a web/internet related application, then the utilitymanagement module 2240 removes web/internet related applications fromthe retrieved image 2270 before configuring the resources 2250. Theutility management module 2240 then applies the modified image 2275 tothe allocated set of resources. Thereafter, the virtual machine isconfigured according to the user's specifications and the virtualmachine begins operating.

At the second stage of operation 2220, a second level of customizationoccurs to an image 2280 retrieved for configuring a second set ofresources 2260 allocated for a second virtual machine of the node. Thissecond level of customization involves modifying the retrieved image2280 by adding additional components not within the retrieved imagebased on the user's virtual machine configuration to produce thecustomized image 2285. For instance, some users may execute, test, ordevelop their proprietary software solutions on the hosted nodes.Accordingly, in some embodiments, the utility management module 2240installs the desired components that were specified and made availableat the time of defining the configuration. The modified image 2285 isthen applied to the allocated set of resources 2260 to create andconfigure the resources per the user's specification in the secondvirtual machine configuration. In some other embodiments, the utilitymanagement module first retrieves an OS image from an OS image store.Then, based on the user configuration, the utility management moduleretrieves additional software application images from an applicationsimage store. In this manner, the utility management module modularlybuilds the user specified configuration, one software component at atime and modifying parameters of each component when necessary.

At the third stage of operation 2230, a third level of customizationoccurs to an image 2290 retrieved for configuring a third set ofresources 2265 allocated for a third virtual machine of the node. Thisthird level of customization involves modifying values for parametersand settings of the operating system or applications within theretrieved image 2270 to produce the customized image 2295. Thesesettings are often unique to each user and may require that themodifications be made at the time of installation or after installationof the image. Such values cannot be known or accounted for when creatingand storing the images within the image store. Typically, these valuesare manually changed by a system operator after the configuration isapplied to the set of resources. However, some embodiments of theutility management module 2240 automate such modifications by applyingthe changes directly to the retrieved image or by applying the changesto a running image that has been previously applied to the allocated setof resources for the configuration. The modified image 2295 is thenapplied to the allocated set of resources 2260 to create and configurethe resources per the user's specification in the second virtual machineconfiguration.

Some examples of custom parameters include: (1) a user adding a webserver through the front-end logic and specifying custom IP addressesfor the server or (2) a user adding a data server through the front-endlogic and specifying custom queries to run with the data server. Itshould be apparent to one of ordinary skill in the art that each of thevarious customizations illustrated above can be performed to a singleretrieved image for a particular virtual machine configuration.Moreover, the three customizations illustrated above are not intended tobe a comprehensive set of customizations, but rather an illustrative setof customizations performed by the utility management module of someembodiments.

FIG. 23 presents a process 2300 implemented by a utility managementmodule of some embodiments to customize an image in accordance with FIG.22. The process 2300 begins when the utility management module receives(at 2310) a user specified virtual machine configuration to install. Theprocess retrieves (at 2320) an OS image from an OS image store thatclosely matches the OS requested within the user specified virtualmachine configuration. The process then determines (at 2330) if theretrieved OS image requires modification based on the parametersspecified within the configuration. As described above, thisdetermination is based on whether operational parameters for the OS havebeen set by the user (e.g., IP address, hostnames, proxy serverparameters, etc.).

When the process determines that no modifications are to be made to theOS image, the process proceeds to step 2360. Otherwise, the processcustomizes (at 2340) the OS parameters based on the user specifiedconfiguration. Accordingly, the utility management module configures thevarious operational parameters for the OS. Additionally, the userspecified configuration may specify the deletion of various OScomponents. For example, a particular OS may include a web serverapplication. However, the user may opt to delete the included web serverapplication and instead install a proprietary or other web serverapplication. Therefore, the process deletes (at 2350) any extraneouscomponents from the configuration.

At 2360, the process determines whether additional components are to beinstalled in conjunction with the OS image. When no further additionalcomponents are required, the process proceeds to install (at 2395) theretrieved and modified images for the virtual machine. Otherwise, theprocess retrieves (at 2370) the desired application images from theapplication image store.

The process performs an additional determination (at 2380) as to whetherthe application images are to be modified. When necessary, the processcustomizes (at 2390) the application parameters according the userspecified configuration. For instance, certain applications may requireaccess to data stored in a data server that is installed on a differentnode. Accordingly, the process configures the application with thenecessary authentication and authorization parameters. The process theninstalls (at 2395) the retrieved and modified images and ends.

C. Modifying Configurations

Should any of configurations be modified after commencement ofoperations, some embodiments of the utility management module providethe functionality to further modify the configurations. As mentionedabove, the utility management module is able to access the file systemsof any of virtual machines. Therefore, the utility management module ofsome embodiments makes the changes during operation of the virtualmachine. Alternatively, the utility management module of someembodiments suspends operation of the virtual machine until the changesare complete. In still other embodiments, the utility management modulecreates a duplicate image on which to apply the changes. Once thechanges are complete, the modified configuration begins operations andthe unmodified configuration is suspended resulting in little to nodowntime to the user's virtual machine.

FIG. 24 illustrates an exemplary set of operations performed by autility management module of some embodiments to modify a configuration.Specifically, FIG. 24 illustrates the utility management module resizingdisk storage that was previously allocated to a virtual machineconfiguration and configuring an image on the modified resource.

On lines 1-2, the utility management module identifies the hostname ofthe virtual machine to be modified and determines the OS image from theOS image store to install. A password is set for security in line 3. Online 4, the utility management module executes a script that uses SSH tolog into the virtual machine where the image will be installed andremoves unexecuted tasks for the last job that was to be processed. Theutility management module creates (e.g., partitions and formats) a locallogical file system to temporarily store the virtual machine image asthe image is built by the utility management module. In creating thelocal file system, the utility management module specifies variousparameters that include the user account number (16323), sessionidentification (730), job number (5756), and the size of the diskstorage (60 Gb) to allocate to the modified virtual machine. The utilitymanagement module then attaches the block device to the utilitymanagement module as a secondary disk at line 7.

On line 8, the utility management module performs an install script thatinstalls the modified configuration with a user-specified configuration.Then on lines 9-19, the utility management module copies the contents ofthe local file system to the file system of the virtual machine in orderto instantiate the virtual machine with the resized storage. Lines 12-19illustrate various statistical information that is generated during thetransfer of the virtual machine from the utility management module'slocal file system to the file system of the virtual machine. In someembodiments, the collected statistics are reported back to thehypervisor management module.

The image is then made bootable through a set of system level commandsat lines 20-22. Partition tables are updated at line 23. The file systemthat now contains the transferred image is mounted to the virtualmachine at line 24. The utility management module performs consistencychecks to ensure the proper installation of the image on the disk andcollects additional statistics for reporting at lines 25-43. Once theresources have been resized and the installation of the image iscomplete, some embodiments then modify the configuration files of thehypervisor running on the node on which the virtual machine wasinstalled. The utility management module modifies the configurationfiles so that the hypervisor is able to recognize the newly installedvirtual machine.

It should be apparent to one of ordinary skill in the art that the setof operations presented in FIG. 24 is meant to provide an exemplary setof operations performed by the utility management module in order tomodify a configuration. Additional operations may be performed inconjunction with or instead of the enumerated set of operations in orderto produce the modified configuration.

The utility management module of some embodiments automatically modifiesan existing virtual machine based on one or more performance thresholdsspecified for the virtual machine. In some embodiments, when a userdefines a virtual machine configuration, the user defines each of thehardware resources and software resources that form the overall virtualmachine. For instance, a user specifies an amount of memory andprocessing resources to be used in conjunction with web servers,application servers, databases, etc. as software resources of a virtualmachine. Then for each of the specified resources, the user may specifyone or more thresholds. Based on whether the specified thresholds aremet or exceeded during runtime operation of the virtual machine, theutility management module of some embodiments automatically up-scales ordown-scales the various resources of the virtual machine.

To determine whether to up-scale or down-scale resources, the hostingsystem monitors the usage of the virtual machine resources in order todetermine which thresholds are met or exceeded. As noted above, thehardware resources of each node, and more particularly the resources ofeach virtual machine, may be monitored by the hypervisor managementmodule or by the utility management module.

FIG. 25 conceptually illustrates thresholds for various resources of arunning virtual machine 2510 that are used by a utility managementmodule 2515 of some embodiments to automatically scale the virtualmachine resources. FIG. 25 illustrates three thresholds 2520-2540 forthree separate resources 2550-2570 of the virtual machine 2510: (1) athreshold 2520 specified for allocated processor resources 2550 of thevirtual machine 2510, (2) a threshold 2530 specified for allocatedmemory resources 2560 of the virtual machine 2510, and (3) a threshold2540 specified for performance (e.g., average latency for executing userrequests) of a software resource 2570 of the virtual machine 2510.Specifically, the figure illustrates usage of the three resources2550-2570 at a specific instance in time during the runtime operation ofthe virtual machine 2510.

As shown, resources 2550 and 2570 remain within the acceptable bounds oftheir respective thresholds 2520 and 2540. However, usage of the memoryresource 2560 exceeds the user specified threshold 2530. In some suchinstances, the utility management module 2515 automatically up-scalesthe memory resource 2560 to increase the amount of memory that isallocated to the virtual machine 2510. By having the utility managementmodule 2515 automatically up-scale the allocated memory, the virtualmachine 2510 avoids performance issues related to exceeding the memoryresource 2560. Some such performance issues include excessive diskpaging or memory thrashing. It should be apparent to one of ordinaryskill in the art that though the thresholds of FIG. 25 are shown areupper bound limits, users may also specify lower bound thresholds thewhen met cause the utility management module 2515 to down-scale one ormore resources. For example, the utility management module 2515decreases the amount of allocated memory for the virtual machine 2510.In this manner, the hosting system automatically guarantees performanceof a user's virtual machine (e.g., by up-scaling resources) withouthaving the user continually monitor and adjust the resources of his orher hosted virtual machine. Additionally, the hosting systemautomatically reduces the cost associated with hosting a virtual machinein a hosting system by automatically downscaling resources of the user'svirtual machine at times when the resources of the virtual machine areunder-utilized or demand fails to meet expected usage levels.

To up-scale a resource, the utility management module increases theamount of an already allocated resource. Such up-scaling of a resourceincludes horizontally up-scaling the resource and vertically up-scalingthe resource. The utility management module of some embodimentshorizontally up-scales a resource by replicating the resource such thata single resource performing a particular task is duplicated to includea second such resource that also performs the same task. In someembodiments, horizontally up-scaling a resource includes replicating asoftware resource of the virtual machine to provide a second instance ofthe software resource that processes in parallel those operations thatwere previously processed by only a first instance of the softwareresource.

For example, prior to a horizontal up-scaling of a software resource, avirtual machine operates using a single web server. The virtual machineis allocated 1 GHz of processing cycles and 1 Gb of RAM and theprocessing cycles and memory are only 40% utilized during operating ofthe virtual machine. However, the reason for the 40% utilization is thatthe web server is over-burdened and therefore causes a bottleneck in theoperation of the virtual machine. The utility management modulehorizontally scales the virtual machine by adding a second web serverwhere 500 MHz of processing cycles and 500 Mb of RAM are allocated tothe first web server and the other 500 MHz of processing cycles and 500Mb of RAM are allocated to the second web server in order to cure thebottleneck.

The utility management module of some embodiments vertically up-scales aresource by increasing the allotment of the resource. In someembodiments, vertically up-scaling a resource includes increasing anamount of a hardware resource that has been allocated to a virtualmachine. In some such embodiments, the hypervisor management module inconjunction with the utility management module allocates additionalprocessing power, memory, storage, or network resources to verticallyup-scale the resource.

For example, prior to a vertical modification, a virtual machineoperates using a single web server. The virtual machine is allocated 1GHz of processing cycles and 500 Mb of RAM where the RAM is fullyutilized resulting in page faults and more frequent access to the slowerblock device. The hypervisor management module in conjunction with theutility management module vertically up-scales the virtual machine byadding an additional 500 Mb of RAM to the virtual machine. Theadditional memory results in fewer page faults and thus faster overallperformance.

In some embodiments, horizontally or vertically down-scaling a resourceis performed when excessive resources are allocated to the virtualmachine. In a hosting system, such excess allocation results in the userbeing charged for resources that are not being used. Therefore, byspecifying down-scaling thresholds, the utility management module canautomatically release the unused resources and thus prevent the userfrom being overcharged when demand for the resources are lower thanexpected. In some embodiments, horizontally down-scaling a resourceinvolves the utility management module removing a second resource of avirtual machine that redundantly performs a task already performed by afirst resource of the virtual machine. In some embodiments, verticallydown-scaling a resource involves the utility management moduledecreasing an amount of a resource allocated to the virtual machine.

FIG. 26 presents a process 2600 performed by a utility management moduleof some embodiments to automatically scale a user virtual machine whenone or more resource thresholds for the virtual machine have been met.The process 2600 begins by identifying (at 2610) a virtual machineresource with a specified resource threshold that is met at someinstance in time during runtime operation of the virtual machine. Insome embodiments, the utility management module directly monitors theresources of the virtual machine that a user has specified with thethreshold. In other embodiments, the hypervisor management modulemonitors the virtual machines and the resources of the virtual machineand reports any threshold that is met to the utility management module.

After identifying the resource, the process then determines (at 2620)whether the resource is one that the user desires to be automaticallyscaled. Should the user not desire the resource to be automaticallyscaled, the process ends. In some such instances, the user may benotified via an email, text message, or through the front-end logic ofthe threshold being met so that the user can take manual action toremedy the situation. Should the user desire the resource to beautomatically scaled, the process then determines (at 2630) whether toup-scale or down-scale the resource. In some embodiments, thisdetermination is based on whether the threshold that is met is an upperbound threshold that specifies when to up-scale the resource or a lowerbound threshold that specifies when to down-scale the resource.

The process up-scales (at 2640) the resource by using the utilitymanagement module of some embodiments to horizontally up-scale theresource or vertically up-scale the resource as described above.Similarly, the process down-scales (at 2635) the resource by using theutility management module of some embodiments to horizontally down-scalethe resource or vertically down-scale the resource as described above.After scaling the resource, the process notifies other components of thehosting system of the changes to the virtual machine. These componentsmay include the hypervisor management module of some embodiments, thefront-end logic of the hosting system that provides the graphical viewof the virtual machine to the user, and the hypervisor on the node thatmanages the resource allocation to the virtual machine. Once themodifications to the virtual machine are complete, the threshold willeither be reset or the resource allocation will be such that thethreshold is no longer met and the process ends.

D. Optimizing Configuration Install

In some embodiments, the utility management module optimizes virtualmachine installation and configuration by minimizing the usage of thenode's resources. FIG. 27 conceptually illustrates an example ofoptimizing a virtual machine installation and configuration inaccordance with some embodiments of the embodiment.

FIG. 27 illustrates a node 2705 that is to host two separate virtualmachines 2710 and 2720 and a utility management module 2730 thatinstalls and configures each of the virtual machines 2710 and 2720. Tooptimize the virtual machine installation and configuration, the utilitymanagement module 2730 identifies that the multiple virtual machines2710 and 2720 are configured with the same operating system (e.g.,Windows Server 2003). Rather than install a separate version of theoperating system for each virtual machine, the utility management module2730 creates a shared partition 2740 for some or all of the operatingsystem files. Then, the utility management module 2730 installs thenon-shareable files for the virtual machines 2710 and 2720 in theirseparate partitions and configures the virtual machines 2710 and 2720 toaccess the shared partition 2740 for the shareable files. In someembodiments, the shared files include static operating system files(e.g., network stacks) that are not customized for the virtual machineconfigurations.

As a result of this optimized virtual machine installation andconfiguration, the disk space usage on the node is minimized.Additionally, the users who have specified the virtual machines needallocate a smaller amount of their hosting resources to the operatingsystem of their configuration. The unused disk space may then beallocated to other applications or data storage. It should be apparentto one of ordinary skill in the art that such an optimized imageinstallation applies to any application program. For example, ratherthan create separate installs for the same application program for eachvirtual machine, some embodiments create a single base install of anapplication program in a shared partition of the block device that eachof the virtual machines may access and saved data or other virtualmachine specific files can be stored locally within the virtualmachine's own disk partition that is not the shared partition.

Additionally, it should be apparent to one of ordinary skill in the artthat certain such optimizations may be carried to other node resources.For example, system memory (e.g., random access memory (RAM)) may storeshared operating system files that are required at runtime. In some suchembodiments, the utility management module may share this portion ofmemory amongst multiple virtual machines.

E. Secure Deletion of Virtual Machines

By accessing the file systems within the utility management module's ownnode, the utility management modules of some embodiments provideautomated secure deletion for a virtual server that is removed from thenode's resources. For instance, rather than simply reallocating theresources to a different virtual machine, the utility management moduleof some embodiments writes random data to the file system (i.e.,overwrites the values stored on a physical storage) such that subsequentconfigurations on the file system will be unable to extract any of thepreviously stored data. Alternatively, some embodiments perform thesecure deletion by “zeroing out” the bits of the previously allocatedfile system.

FIG. 28 presents a process 2800 for securely deleting a virtual machineautomatically using a utility management module of some embodiments. Theprocess 2800 begins when a utility management module operating at a nodeof a hosting system is directed to securely delete a virtual machine onthe same node. Specifically, the process identifies the virtual machine(at 2810) to be deleted and the corresponding set of resources allocatedto the virtual machine.

The process then stops (at 2820) the virtual machine's operationsthrough a set of override commands that turn-off the virtual machine.The process accesses (at 2830) the block device of the virtual machineand performs (at 2840) a secure deletion of the virtual machineconfiguration and data. To perform the secure deletion, the processwrites sets of random data to the disk storage (i.e., block device)ensuring that none of the customer's data remains on the disk storagebefore the disk storage is allocated to another customer.

The process removes (at 2850) other resources allocated to the virtualmachine and deletes the virtual machine from the hypervisor controllingthe node's resources. The process then reports (at 2860) the releasedset of resources back to the hypervisor management module and theprocess ends. The released resources are mapped to the pool of availablehosting resources of the hosting system provider where the resources maybe allocated to other virtual machines hosted by the hosting systemprovider.

Without the secure deletion performed by some embodiments, the virtualmachine is removed, but the data physically remains on the disk device.The disk storage storing the previous user's data may then be allocatedfor use by a second virtual machine. Thereafter, the newly allocatedsecond virtual machine may scan the block device providing the diskstorage and extract the undeleted data. However, by using the automatedsecure deletion functionality provided by the utility management moduleof some embodiments, the data is never accessible by subsequent virtualmachines ensuring customer privacy.

F. Virtual Machine Migration

In addition to accessing file systems within the utility managementmodule's own node and the image store, some embodiments allow utilitymanagement modules access to each other. FIG. 29 illustrates theinterfacing between utility management modules in order to migrate avirtual machine configuration from one node to another. In this figure,the grid of nodes includes three nodes 2910, 2920, and 2930 with theutility management module 2940 of node 2910 communicating with theutility management module 2950 of node 2930 in order to reallocate andreconfigure configuration 2960 onto node 2930.

In some embodiments, the hypervisor management module 2905 initiates themigration. To do so, the hypervisor management module 2905 firstinstructs a hypervisor of the destination node 2930 to allocatesufficient resources for the virtual machine that is to be migrated.These instructions may include allocating processing resources, memory,etc. The hypervisor management module 2905 then instructs the utilitymanagement module 2940 to transfer the virtual machine 2960 from node2910 to node 2930. The utility management module 2940 initiates thetransfer by communicating with the utility management module 2950 ofnode 2930 and the configuration for the virtual machine 2960 istransferred from node 2910 to node 2930 where it is reconfigured tooperate as virtual machine 2970.

FIG. 30 presents a process 3000 for migrating a virtual machine from afirst node to a second node in a grid of nodes in accordance with someembodiments. The process 3000 begins by determining (at 3010) one ormore configurations for virtual machines at a particular node that areto be migrated over to a different node. In some embodiments,determining the configuration involves identifying (at 3020) thesoftware resources and particular configuration parameters for thevirtual machines. Additionally, in determining the configuration, theprocess also identifies (at 3020) the physical resources allocated toeach of the virtual machines.

The process then attempts to locate a suitable host for theconfigurations by identifying (at 3030) other nodes with sufficientavailable resources to host the migrating virtual machines. This mayoccur via services provided by the hypervisor management module of someembodiments. The hypervisor management module retains a complete view ofthe available and unavailable resources of the grid of nodes and istherefore able to quickly identify the replacement nodes. In someembodiments, one or more different nodes are identified for each virtualmachine. The process allocates (at 3040) resources of the identifiednodes in a manner that maximizes the hosting capabilities of the nodes.The configuration for each particular virtual machine is then passed (at3050) to the corresponding new host node in order to configure (at 3060)the allocated resources.

In order to complete the migration, the process stops (at 3070) theoperation of the previous virtual machines and deletes the previousvirtual machines. At the same time, the process commences (at 3080) theoperation of the newly configured nodes completing the migrationprocess.

In some embodiments, such virtual machine migration is useful in orderto avoid downtime when a node within the grid of nodes ismalfunctioning. Rather than suspend the operation of all virtualmachines on that node, some embodiments identify other nodes within thegrid of nodes with sufficient available resources to temporarily orpermanently host the one or more virtual machine configurations on thefailed node. In some embodiments, the utility management modulesleverage the functionality of the hypervisor management module toidentify the nodes with the available resources and to communicate withthe other nodes in order to allocate the resources necessary for themigration.

Such virtual machine migration is also useful when upgrading a node withadditional functional components (i.e., additional memory or diskstorage) or upgrading a configuration where the node currently hostingthe configuration runs out of available resources to satisfy theupgrade. For instance, a particular configuration on a first node isallocated two gigabytes of memory. However, at a subsequent time, theuser modifies the configuration to request four gigabytes of memory. Ifthe node on which the configuration currently operates only has anadditional one gigabyte of free memory, then either a portion of theconfiguration or the entire configuration must be migrated to adifferent node with the necessary free memory resources.

V. Computer System

Many of the above-described processes and modules are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable hardware medium (also referred to as computerreadable medium or a computer readable storage medium). When theseinstructions are executed by one or more computational element(s) (suchas processors or other computational elements like ASICs and FPGAs),they cause the computational element(s) to perform the actions indicatedin the instructions. Computer is meant in its broadest sense, and caninclude any electronic device with a processor. Examples of computerreadable hardware media are described below in greater detail.

In this specification, the term “software” is meant in its broadestsense. It can include firmware residing in read-only memory orapplications stored in magnetic storage which can be read into memoryfor processing by a processor. Also, in some embodiments, multiplesoftware inventions can be implemented as sub-parts of a larger programwhile remaining distinct software inventions. In some embodiments,multiple software inventions can also be implemented as separateprograms. Finally, any combination of separate programs that togetherimplement a software invention described here is within the scope of theinvention. In some embodiments, the software programs when installed tooperate on one or more computer systems define one or more specificmachine implementations that execute and perform the operations of thesoftware programs.

FIG. 31 illustrates a computer system with which some embodiments of theinvention are implemented. Such a computer system includes various typesof computer readable hardware media and interfaces for various othertypes of computer readable hardware media that implement the variousprocesses and modules described above (e.g., hypervisor managementmodule and utility management modules). Computer system 3100 includes abus 3105, a processor 3110, a system memory 3115, a read-only memory3120, a permanent storage device 3125, input devices 3130, and outputdevices 3135.

The bus 3105 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of thecomputer system 3100. For instance, the bus 3105 communicativelyconnects the processor 3110 with the read-only memory 3120, the systemmemory 3115, and the permanent storage device 3125. From these variousmemory units, the processor 3110 retrieves instructions to execute anddata to process in order to execute the processes of the invention. Theprocessor 3110 is a processing device such as electronic circuitry.

The read-only-memory (ROM) 3120 stores static data and instructions thatare needed by the processor 3110 and other modules of the computersystem. The permanent storage device 3125, on the other hand, is aread-and-write memory device. This device is a non-volatile memory unitthat stores instructions and data even when the computer system 3100 isoff. Some embodiments of the invention use a mass-storage device (suchas a magnetic or optical disk and its corresponding disk drive) as thepermanent storage device 3125.

Some other embodiments use a removable storage device (such as a floppydisk, flash drive, read-only compact discs (CD-ROM), read-only digitalversatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), or ZIP® disks) asthe permanent storage device. Like the permanent storage device 3125,the system memory 3115 is a read-and-write memory device. However,unlike storage device 3125, the system memory is a volatileread-and-write memory, such a random access memory (RAM). The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 3115, the permanent storage device 3125, and/or theread-only memory 3120. Some other examples of such computer readablehardware media include rewritable compact discs (CD-RW), a variety ofrecordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flashmemory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magneticand/or solid state hard drives, read-only and recordable blu-ray discs,any other optical or magnetic media, and floppy disks.

The ROM 3120, system memory 3115, permanent storage device 3125, andother above enumerated physical storage devices of the computer system3100, collectively or separately, represent the computer readablehardware medium of each of the hypervisor management module of someembodiments, utility management modules of some embodiments, and thenodes of some embodiments.

In some embodiments, the computer readable hardware media stores theprocesses and executable instructions that implement the functionalityof the hypervisor management module. For example, the computer readablehardware media stores the processes for generating the provisioningmessages and the control messages used by the hypervisor managementmodule in order to control operation of the grid of nodes.

Similarly, in some embodiments, the computer readable hardware mediastores the processes and executable instructions that implement thefunctionality of the utility management module. For example, thecomputer readable hardware media stores the scripts that define thecustomizations that the utility management module performs to aretrieved software image in order to generate a virtual machine that iscustomized according to user specifications.

As used in this specification and any claims of this application, theterm computer readable hardware medium is entirely restricted totangible, physical objects that store information in a form that isreadable by a computer. These terms exclude any wireless signals, wireddownload signals, and any other ephemeral signals. However, such signalsmay define other forms of non-computer readable medium.

The bus 3105 also connects to the input and output devices 3130 and3135. The input devices enable the user to communicate information andselect commands to the computer system. The input devices 3130 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The input devices 3130 also include audio input devices(e.g., microphones, MIDI musical instruments, etc.). The output devices3135 display images generated by the computer system. For instance,these devices display a GUI. The output devices include printers anddisplay devices, such as cathode ray tubes (CRT) or liquid crystaldisplays (LCD).

Finally, as shown in FIG. 31, bus 3105 also couples computer 3100 to anetwork 3165 through a network adapter (not shown). In this manner, thecomputer can be a part of a network of computers (such as a local areanetwork (“LAN”), a wide area network (“WAN”), or an Intranet, or anetwork of networks, such as the internet. For example, the computer3100 may be coupled to a web server (network 3165) so that a web browserexecuting on the computer 3100 can interact with the web server as auser interacts with a GUI that operates in the web browser.

Any or all components of computer system 3100 may be used in conjunctionwith the invention. Though type 1 or type 2 hypervisors are shown insome of the various figures, it should be apparent to one of ordinaryskill in the art that any type of hypervisor may operate with thevarious components of some embodiments, such as the hypervisormanagement module and the utility management module.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. Thus, one of ordinary skill in the artwould understand that the invention is not to be limited by theforegoing illustrative details, but rather is to be defined by theappended claims.

1. For a hypervisor management module that interfaces with a pluralityof resource nodes of a hosting system, each of the resource nodescomprising resources for hosting a plurality of virtual machines, amethod for deploying virtual machines of a plurality of users on thehosting system, the method comprising: from a plurality of hypervisorsthat each operate to virtualize the resources of a resource node,collecting a first set of statistics of the resource nodes on which thehypervisors operate, wherein the plurality of resource nodes of thehosting system comprise at least two different types of hypervisors;from a plurality of utility management modules that each operate as avirtual machine on a resource node, collecting a second set ofstatistics of the resource nodes on which the utility management modulesoperate, wherein each utility management module is a virtual machineoperating on the resources of a particular resource node to deploy andconfigure user-specified virtual machines on the particular resourcenode, wherein the hypervisor management module collects statistics fromboth the hypervisor and the utility management module of at least oneresource node; based on the first and second sets of collectedstatistics, generating a mapping of resource usage and resourcesavailable on the plurality of resource nodes; receiving a modificationto a particular configuration of a particular virtual machine hosted bya first node of the plurality of resource nodes, wherein the first nodedoes not have sufficient available resources to host the particularmodified virtual machine; based on the generated mapping, determining adifferent node from the plurality of resource nodes to host the modifiedconfiguration for the virtual machine by (i) identifying a hypervisoroperating on the first node and (ii) identifying a second node havingsufficient available resources to host the particular modified virtualmachine and having the same type of hypervisor as the first node; andmigrating the particular configuration from the first node to the secondnode of the plurality of resource nodes by installing the modifiedconfiguration to the second node.
 2. The method of claim 1, wherein thesets of statistics comprise usage statistics for the resources of theplurality of resource nodes, the usage statistics comprising usage of atleast one of processing resources, memory, disk space, networkinput/output bandwidth, and software performance of the plurality ofresource nodes.
 3. The method of claim 1, wherein the sets of statisticscomprise data regarding a number of virtual machines hosted on theresource nodes and the configurations of the virtual machines deployedto the resource nodes.
 4. The method of claim 1, wherein the hypervisorof a particular node is for facilitating access between a plurality ofvirtual machines of the particular node and resources of the particularnode.
 5. The method of claim 4 further comprising normalizing thestatistics collected from different hypervisors to produce uniformlyformatted statistics by converting different data units of each of thestatistics from the different hypervisors into a single data unit. 6.The method of claim 1 further comprising directly interfacing withhardware resources of at least one node and collecting a third set ofstatistics directly from the hardware resources.
 7. The method of claim1, wherein migrating the particular configuration comprises allocating aset of resources from the second node to host the modified configurationof the particular virtual machine and deleting the particular virtualmachine from the first node.
 8. The method of claim 1, whereincollecting the sets of statistics for facilitating deployment comprisesdetermining availability of resources for each of the nodes in theplurality of resource nodes.
 9. The method of claim 1 furthercomprising: updating the collected statistics when the modifiedconfiguration of the virtual machine is migrated; and based on theupdated statistics, generating an updated mapping of resource usage andresources available on the plurality of nodes.
 10. The method of claim1, wherein each of a plurality of the resource nodes is a physicalcomputer comprising a set of processors and memory that are sharedbetween the virtual machines hosted on the resource node.
 11. A hostingsystem for hosting a plurality of virtual machine configurations for aplurality of users, the hosting system comprising: a plurality of nodes,each particular node comprising physical resources for hosting at leastone user-specified virtual machine; and a hypervisor management modulefor managing the deployment of user-specified virtual machines on theplurality of nodes of the hosting system, the hypervisor managementmodule comprising: a statistics collector for (i) collecting, from aplurality of hypervisors that each operate to virtualize the resourcesof a node, a first set of statistics of the nodes on which thehypervisors operate, wherein the plurality of nodes of the hostingsystem comprise at least two different types of hypervisors, and (ii)collecting, from a plurality of utility management modules that eachoperate as a virtual machine on a node, a second set of statistics ofthe nodes on which the utility management modules operate, wherein eachutility management module is a virtual machine operating on theresources of a particular node to deploy and configure user-specifiedvirtual machines on the particular node, wherein the statisticscollector collects statistics from both the hypervisor and the utilitymanagement module of at least one node; a mapping generator forgenerating a mapping of resource usage and resources available on theplurality of nodes based on the first and second sets of collectedstatistics; a deployment scheduler for (i) receiving a modification to aparticular configuration of a particular virtual machine hosted by afirst node of the plurality of nodes, wherein the first node does nothave sufficient available resources to host the particular modifiedvirtual machine, and (ii) determining, based on the generated mapping, adifferent node from the plurality of nodes to host the modifiedconfiguration for the virtual machine by identifying a hypervisoroperating on the first node and identifying a second node havingsufficient available resources to host the particular modified virtualmachine and having the same type of hypervisor as the first node; and adeployment controller for managing the migration of the particularconfiguration from the first node to the second node of the plurality ofnodes by installing the modified configuration to the second node. 12.The hosting system of claim 11, wherein the plurality of nodes comprisesa plurality of server computer systems, each particular server computersystem comprising a set of processing resources, memory, and diskstorage allocated to a plurality of virtual machines that are hosted bythe particular server computer system.
 13. The hosting system of claim11, wherein the hypervisor management module further comprises aresource allocation module for allocating resources to the virtualmachine configurations.
 14. The hosting system of claim 11 furthercomprising a front end server for providing a graphical user interfacethrough which (i) a portion of the statistics are presented to the userand (ii) the user specifies the modification to the virtual machineconfiguration.
 15. A non-transitory computer readable storage mediumstoring a hypervisor management module that interfaces with a pluralityof nodes of a hosting system, the nodes comprising resources for hostingvirtual machines of a plurality of different users, the hypervisormanagement module for execution by at least one processor of the hostingsystem, the hypervisor management module comprising: a set ofinstructions for collecting, from a plurality of hypervisors that eachoperate to virtualize the resources of a node, a first set of statisticsof the nodes on which the hypervisors operate, wherein the plurality ofnodes of the hosting system comprise at least two different types ofhypervisors; a set of instructions for collecting, from a plurality ofutility management modules that each operate as a virtual machine on anode, a second set of statistics of the nodes on which the utilitymanagement modules operate, wherein each utility management module is avirtual machine operating on the resources of a particular node todeploy and configure user-specified virtual machines on the particularnode, wherein the hypervisor management module collects statistics fromboth the hypervisor and the utility management module of at least onenode; a set of instructions for generating, based on the first andsecond sets of collected statistics, a mapping of resource usage andresources available on the plurality of nodes; a set of instructions forreceiving a modification to a particular configuration of a particularvirtual machine hosted by the first node of the plurality of nodes,wherein the first node does not have sufficient available resources tohost the particular modified virtual machine; a set of instructions fordetermining, based on the generated mapping, a different node from theplurality of nodes to host the modified configuration for the virtualmachine by (i) identifying a hypervisor operating on the first node and(ii) identifying a second node having sufficient available resources tohost the particular modified virtual machine and having the same type ofhypervisor as the first node; and a set of instructions for migratingthe particular configuration from the first node to the second node ofthe plurality of resource nodes by installing the modified configurationto the second node.
 16. A method for providing a hypervisor managementmodule for deploying configurations of virtual machines on a hostingsystem comprising a plurality of nodes, each node comprising resourcesfor hosting a plurality of virtual machines and a hypervisor forproviding access to the resources for the virtual machines hosted on thenode, the method comprising: providing a statistics collector for (i)collecting, from the hypervisors of the plurality of nodes, a first setof statistics of the nodes on which the hypervisors operate, wherein theplurality of nodes of the hosting system comprise at least two differenttypes of hypervisors, and (ii) collecting, from a plurality of utilitymanagement modules that each operate as a virtual machine on a node, asecond set of statistics of the nodes on which the utility managementmodules operate, wherein each utility management module is a virtualmachine operating on the resources of a particular node to deploy andconfigure user-specified virtual machines on the particular node,wherein the statistics collector collects statistics from both thehypervisor and the utility management module of at least one node;providing a mapping generator for generating a mapping of resource usageand resources available on the plurality of nodes based on the first andsecond sets of collected statistics; providing a deployment schedulerfor (i) receiving a modification to a particular configuration of aparticular virtual machine hosted by a first node of the plurality ofnodes, wherein the first node does not have sufficient availableresources to host the particular modified virtual machine, and (ii)determining, based on the generated mapping, a different node from theplurality of nodes to host the modified configuration of the virtualmachine by identifying a hypervisor operating on the first node andidentifying a second node having sufficient available resources to hostthe particular modified virtual machine and having the same type ofhypervisor as the first node; and providing a deployment controller formanaging the migration of the particular configuration from the firstnode to the second node of the plurality of nodes by installing themodified configuration to the second node.
 17. The method of claim 16further comprising providing a normalization engine for normalizing thecollected statistics from the hypervisors to produce uniformly formattedstatistics.
 18. The method of claim 16 further comprising: providing adatabase for storing message translations between a normalized set ofmessages and messages for the different types of hypervisors of theplurality of nodes; and providing one or more deployment managerscorresponding to different types of hypervisors, the one or moredeployment managers for (i) using the database of stored messages totranslate normalized control messages to be executed by correspondinghypervisors in order to deploy virtual machines on the nodes and (ii)submitting the translated control messages to the hypervisors, whereinthe deployment controller is further for (i) identifying a deploymentmanager capable of translating a particular control message and (ii)sending the particular control message to the identified deploymentmanager to provide instructions to a hypervisor corresponding to theidentified deployment manager in order to deploy virtual machines on thenodes.
 19. The method of claim 18 further comprising providing a messageprocessing module for (i) receiving, from users through a front-end userinterface, messages specifying logical assignments of virtual machineconfigurations and (ii) passing the messages specifying the logicalassignments of the virtual machine configurations to the deploymentscheduler in order to deploy virtual machines on the nodes.